{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/5-Fine%20Tuning/QLoRA_Tuning_PEFT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j9Y40X0WCY_H"
      },
      "source": [
        "<div align=\"center\">\n",
        "<h1><a href=\"https://github.com/peremartra/Large-Language-Model-Notebooks-Course\">Learn by Doing LLM Projects</a></h1>\n",
        "    <h3>Understand And Apply Large Language Models</h3>\n",
        "    <h2>QLoRA Tuning using PEFT from Hugging Face.</h2>\n",
        "    <h3>Introduction to Quantization & Fine-tune a Quantized Model</h3>\n",
        "    by <b>Pere Martra</b>\n",
        "</div>\n",
        "\n",
        "<br>\n",
        "\n",
        "<div align=\"center\">\n",
        "    &nbsp;\n",
        "    <a target=\"_blank\" href=\"https://www.linkedin.com/in/pere-martra/\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>\n",
        "    \n",
        "</div>\n",
        "\n",
        "<br>\n",
        "<hr>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Brief Introduction to Quantization\n",
        "Quantization is a method that reduces the memory used by Large Language Models. It is achieved by reducing the precisión of the numbers involved in the Model.\n",
        "\n",
        "We transform 32-bit floating-point numbers into 8-bit or 4-bit integers, thereby reducing the memory and computational requirements of the quantized Large Language Model.\n",
        "\n",
        "If you want to load a 7B Model into the memory of a 16Gb GPU you must quantize it.\n",
        "\n"
      ],
      "metadata": {
        "id": "_afIANF7iKXp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Let's see a example of a quantized number.\n",
        "First, I'm going to create a function to quantize and other to unquantize."
      ],
      "metadata": {
        "id": "jyoaSeEAi8_W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Importing necesary linbraries\n",
        "import numpy as np\n",
        "import math\n",
        "import matplotlib.pyplot as plt"
      ],
      "metadata": {
        "id": "yj5-xG8WogNP"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Functions to quantize and unquantize\n",
        "def quantize(value, bits=4):\n",
        "    quantized_value = np.round(value * (2**(bits - 1) - 1))\n",
        "    return int(quantized_value)\n",
        "\n",
        "def unquantize(quantized_value, bits=4):\n",
        "    value = quantized_value / (2**(bits - 1) - 1)\n",
        "    return float(value)"
      ],
      "metadata": {
        "id": "k17vVVU9iKws"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "quant_4 = quantize(0.622, 4)\n",
        "print (quant_4)\n",
        "quant_8 = quantize(0.622, 8)\n",
        "print(quant_8)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XXRJ7mJMlpjA",
        "outputId": "7868bcfa-22c4-4318-dd3f-69d57c47fea2"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4\n",
            "79\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "unquant_4 = unquantize(quant_4, 4)\n",
        "print(unquant_4)\n",
        "unquant_8 = unquantize(quant_8, 8)\n",
        "print(unquant_8)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y50sGnmbmCqv",
        "outputId": "553e7b86-b1d0-485e-a95d-4b31f150eb0e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.5714285714285714\n",
            "0.6220472440944882\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The original numbes was 0.6222 and once unquantized the values are 0.571 for the 4 bit, and 0.622 for the 8 bit quantization.\n",
        "\n",
        "The precision is more or less the same, especially in the case of 8-bit quantization."
      ],
      "metadata": {
        "id": "eyjuQtfFTalb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x = np.linspace(-1, 1, 50)\n",
        "y = [math.cos(val) for val in x]\n",
        "\n",
        "\n",
        "y_quant_8bit = np.array([quantize(val, bits=8) for val in y])\n",
        "y_unquant_8bit = np.array([unquantize(val, bits=8) for val in y_quant_8bit])\n",
        "\n",
        "y_quant_4bit = np.array([quantize(val, bits=4) for val in y])\n",
        "y_unquant_4bit = np.array([unquantize(val, bits=4) for val in y_quant_4bit])\n"
      ],
      "metadata": {
        "id": "KzCAXBmMnNSA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 12))\n",
        "\n",
        "plt.subplot(4, 1, 1)\n",
        "plt.plot(x, y, label=\"Original\")\n",
        "plt.plot(x, y_unquant_8bit, label=\"unquantized_8bit\")\n",
        "plt.plot(x, y_unquant_4bit, label=\"unquantized_4bit\")\n",
        "plt.legend()\n",
        "plt.title(\"Compare Graph\")\n",
        "plt.grid(True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "u35LgstBoaTQ",
        "outputId": "e3ac0afe-9723-4dbb-cd47-9c135ebcf9ec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x1200 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzoAAAEKCAYAAADaVCAwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACFUElEQVR4nOzdd3gU1dfA8e/sZtN7TyAk9A6hJYCFIkWaIBYUlCACgqAgKoqF5qs/FaWoFFGaKGIBQQXpotQk9BZ66Kmk92R33j9iVmICJJDNppzP8+wDO3tn5szJZrNn5s69iqqqKkIIIYQQQghRhWjMHYAQQgghhBBClDUpdIQQQgghhBBVjhQ6QgghhBBCiCpHCh0hhBBCCCFElSOFjhBCCCGEEKLKkUJHCCGEEEIIUeVIoSOEEEIIIYSocqTQEUIIIYQQQlQ5UugIIYQQQgghqhwpdIQQQogKZNiwYdjb25s7DCGEqPSk0BFCiEro/PnzvPDCC9SpUwdra2scHR257777mDt3LpmZmeYOr8IzGAx88803dO/eHXd3d3Q6HZ6envTo0YNFixaRnZ1t7hCFEELcIwtzByCEEKJ01q9fzxNPPIGVlRVDhw6lWbNm5OTksGvXLl5//XVOnDjBokWLzB1mhZWZmcmjjz7Kpk2b6NixI6+99hpeXl4kJCTw119/8eKLLxIaGsrixYvNHaoQQoh7IIWOEEJUIpGRkTz11FP4+/uzfft2fHx8jK+NHTuWc+fOsX79ejNGaDoGg4GcnBysra3vaTuvvPIKmzZtYs6cOYwfP77Qa6+++ipnz55ly5Ytt91GXl4eBoMBS0vLe4pFCCGE6UjXNSGEqEQ+/vhj0tLSWLx4caEip0C9evUKfXnPy8vjvffeo27dulhZWREQEMBbb71VpGtWQEAAffv2ZceOHbRt2xYbGxuaN2/Ojh07AFizZg3NmzfH2tqaNm3acOjQoULrF9xXcuHCBXr27ImdnR2+vr7MmDEDVVULtf3kk0/o2LEjbm5u2NjY0KZNG37++ecix6IoCuPGjeO7776jadOmWFlZsXHjRgCuXbvG8OHD8fLywsrKiqZNm7JkyZI75u/KlSt8/fXXPPzww0WKnAL169fnxRdfND6/ePEiiqLwySefMGfOHGMuT548SU5ODlOmTKFNmzY4OTlhZ2fHAw88wJ9//llomzdvY/bs2fj7+2NjY0OnTp04fvx4sXFcu3aNAQMGYG9vj4eHB6+99hp6vf6OxyiEECKfov73L5AQQogKq2bNmlhZWXH+/PkStR82bBjLly/n8ccfp0uXLoSGhvLNN98wYMAAfvnlF2O7gIAArK2tSUlJ4YUXXsDJyYlPPvmE5ORkFi5cyFtvvWX88v+///0PDw8PTp8+jUajMe7nhx9+wM/Pj/bt2xMcHMzGjRv5/fffeffdd5kxY4ZxX35+fjzyyCM0adKEnJwcVq1aRVhYGL///jt9+vQxtlMUhcaNGxMfH8+4ceNwd3enY8eO+Pj40LZtWxRFYeTIkXh4ePDHH3/w66+/Mnv2bCZMmHDLfCxatIgXXniBb7/9liFDhpQohxcvXqR27do0adKErKwsRo0ahZWVFQMHDsTW1pYWLVrw9NNPU79+fVJTU1m8eDEXLlwgLCyMwMDAQtto3rw5qampjBkzhqysLObOnYtWq+XYsWN4eXkVymXdunUJDg6mXbt2bN26ldWrVzN//nzGjBlToriFEKLaU4UQQlQKycnJKqD279+/RO0PHz6sAuqIESMKLX/ttddUQN2+fbtxmb+/vwqoe/bsMS7btGmTCqg2NjbqpUuXjMu//PJLFVD//PNP47KQkBAVUF966SXjMoPBoPbp00e1tLRU4+LijMszMjIKxZOTk6M2a9ZM7dq1a6HlgKrRaNQTJ04UWv7888+rPj4+anx8fKHlTz31lOrk5FRk+zd75ZVXVEA9fPhwoeXZ2dlqXFyc8XHztiMjI1VAdXR0VGNjYwutl5eXp2ZnZxdalpiYqHp5eanDhw8vsg0bGxv16tWrxuWhoaEqoL7yyivGZQW5nDFjRqHttmrVSm3Tps0tj00IIURh0nVNCCEqiZSUFAAcHBxK1H7Dhg0ATJw4sdDyV199FaDIvTxNmjShQ4cOxufBwcEAdO3alVq1ahVZfuHChSL7HDdunPH/BV3PcnJy2Lp1q3G5jY2N8f+JiYkkJyfzwAMPcPDgwSLb69SpE02aNDE+V1WV1atX069fP1RVJT4+3vjo2bMnycnJxW6nQEEO/zt884YNG/Dw8DA+/P39i6z72GOP4eHhUWiZVqs13qdjMBhISEggLy+Ptm3bFhvHgAEDqFGjhvF5UFAQwcHBxp/VzUaPHl3o+QMPPFBszoUQQhRPBiMQQohKwtHREYDU1NQStb906RIajYZ69eoVWu7t7Y2zszOXLl0qtPzmYgbAyckJyO9qVtzyxMTEQss1Gg116tQptKxBgwZAftetAr///jv/93//x+HDhwvdK6QoSpFjqF27dqHncXFxJCUlsWjRoluOLBcbG1vscvi3SExLSyu0/L777jMOQDBz5kx27959x1gKLF++nE8//ZRTp06Rm5t72/b169cvsqxBgwb8+OOPhZZZW1sXKapcXFyK5FwIIcStSaEjhBCVhKOjI76+vre8ef1WiisgiqPVaku1XL2LWzx37tzJI488woMPPsj8+fPx8fFBp9OxdOlSVq5cWaT9zVd/IP+qCcAzzzxDSEhIsfto0aLFLfffqFEjAI4fP07Lli2Nyz08POjWrRsA3377bbHr/jeWgrbDhg1jwIABvP7663h6eqLVavnf//5X4vuoinOrnAshhCg5KXSEEKIS6du3L4sWLWLv3r2FupkVx9/fH4PBwNmzZ2ncuLFxeUxMDElJScV2z7oXBoOBCxcuGK/iAJw5cwbIH+wAYPXq1VhbW7Np0yasrKyM7ZYuXVqifXh4eODg4IBerzcWJqXRq1cvtFot3333XYkHI7idn3/+mTp16rBmzZpCBeXUqVOLbX/27Nkiy86cOWPMjxBCiLIj9+gIIUQlMmnSJOzs7BgxYgQxMTFFXj9//jxz584FoHfv3gDMmTOnUJtZs2YBFBrhrKx88cUXxv+rqsoXX3yBTqfjoYceAvKvVCiKUmiY5IsXL7J27doSbV+r1fLYY4+xevXqYq9sxcXF3Xb9WrVqMXz4cP74449Csd6sNFeqCq683LxOaGgoe/fuLbb92rVruXbtmvF5WFgYoaGh9OrVq8T7FEIIUTJyRUcIISqRunXrsnLlSgYNGkTjxo0ZOnQozZo1Iycnhz179vDTTz8xbNgwAFq2bElISAiLFi0iKSmJTp06ERYWxvLlyxkwYABdunQp09isra3ZuHEjISEhBAcH88cff7B+/Xreeust4/0mffr0YdasWTz88MMMHjyY2NhY5s2bR7169Th69GiJ9vPhhx/y559/EhwczMiRI2nSpAkJCQkcPHiQrVu3kpCQcNv158yZQ2RkJC+99BKrVq2iX79+eHp6Eh8fz+7du/ntt99o2LBhiWLp27cva9as4dFHH6VPnz5ERkaycOFCmjRpUuQ+IMif5+j+++9nzJgxZGdnM2fOHNzc3Jg0aVKJ9ieEEKLkpNARQohK5pFHHuHo0aPMnDmTdevWsWDBAqysrGjRogWffvopI0eONLb9+uuvqVOnDsuWLeOXX37B29ubyZMn37Jr1b3QarVs3LiRMWPG8Prrr+Pg4MDUqVOZMmWKsU3Xrl1ZvHgxH374IRMmTKB27dp89NFHXLx4scSFjpeXF2FhYcyYMYM1a9Ywf/583NzcaNq0KR999NEd17e1tWXjxo2sWLGCFStW8PHHH5OSkoKzszMtW7Zk/vz5t7z/57+GDRtGdHQ0X375JZs2baJJkyZ8++23/PTTT8bJVm82dOhQNBoNc+bMITY2lqCgIL744otiJ38VQghxb2TCUCGEEPds2LBh/Pzzz8VexRD/Thg6c+ZMXnvtNXOHI4QQ1YLcoyOEEEIIIYSocqTQEUIIIYQQQlQ5UugIIYQQQgghqhy5R0cIIYQQQghR5cgVHSGEEEIIIUSVUymGlzYYDFy/fh0HB4dCM08LIYQQQgghqhdVVUlNTcXX1xeN5tbXbSpFoXP9+nX8/PzMHYYQQgghhBCigrhy5Qo1a9a85euVotBxcHAA8g/G0dHRrLHk5uayefNmevTogU6nM2ssVZHk17Qkv6Yl+TUtya9pSX5NS/JrWpJf06po+U1JScHPz89YI9xKpSh0CrqrOTo6VohCx9bWFkdHxwrxg65qJL+mJfk1LcmvaUl+TUvya1qSX9OS/JpWRc3vnW5pkcEIhBBCCCGEEFWOFDpCCCGEEEKIKkcKHSGEEEIIIUSVU+pC5++//6Zfv374+vqiKApr16694zo7duygdevWWFlZUa9ePZYtW3YXoQohhBBCCCFEyZS60ElPT6dly5bMmzevRO0jIyPp06cPXbp04fDhw0yYMIERI0awadOmUgcrhBBCCCGEECVR6lHXevXqRa9evUrcfuHChdSuXZtPP/0UgMaNG7Nr1y5mz55Nz549S7t7IYQoYve13RyOO2zuMNDr9ZzLPMelo5fQarXmDsc8cjMhJQrSYsCgL/SSioqqgsGgoldV9AYVg6qiN/DPv/nLVfVWG1dJT09n/8qlQNGRdhRAo4BWo6DRKGiVf//VahQ0Cmg0Ckox62LtBI6+YOsK1XRi6or0/rWzsOOJhk9gp7MzaxxCiMrN5MNL7927l27duhVa1rNnTyZMmHDLdbKzs8nOzjY+T0lJAfKHtsvNzTVJnCVVsH9zx1FVSX5NqyrmNzk7mXHbx5FnyDN3KEZ/Hv/T3CFUXXf63qsC+n8epZECxN5VRFVORXn/ZuRkMLL5SHOHUWaq4udvRSL5Na2Klt+SxmHyQic6OhovL69Cy7y8vEhJSSEzMxMbG5si6/zvf/9j+vTpRZZv3rwZW1tbk8VaGlu2bDF3CFWa5Ne0qlJ+T+acJM+Qh71iT1NdU3OHUyXkqpCVB1n6/EeuAXL0oDFk42hIwUlNwo0UXEnBQilaUaSqttxQHcnm1nMtKKgoyj/9p//5V1Hyr8rc7fUU9eaHetO/Bf+/xZY1qDgp6biRjFYxFHk9RbXlhupEouJEisaJTI09FloFSw1YasFaCzba/P/LCD/3LsGQwNm8s2w4uYEaV2qYO5wyV5U+fysiya9pVZT8ZmRklKhdhZwwdPLkyUycONH4vGD20x49elSICUO3bNlC9+7dK9SESVWF5Ne0qmJ+j+8/DmegV71eTG432ayxVMj8pkajXD8IWcnGRaqqkpKdR0JaDjfS8x83/z8z99/ixYVUAjXnaKU5h6+SUHTz2HLGoiFXbJsQ59ScdLeWeLp4EmxviaO1BTaWWuwsLbC11P7zsMBGp8FCW/qS4F7yq6oq2XkG0nP0ZOTkkZGtJyNXT0aOnvTsPBIzcrmRnI7FjQicE49SI+0EdbIjqKVeBxKBa8ZtZak6jqm1OWSozznVF8M/5Y1GUXC11eFqb4mb3b8PV3tLXG0t0d18zBotqntD8GwC2orxXqko79/LqZcZ8NsArqnX6NKjCzYWRU+IVkYVJb9VleTXtCpafgt6e92JyQsdb29vYmJiCi2LiYnB0dGx2Ks5AFZWVlhZWRVZrtPpKkRyoWLFUhVJfk2rKuV3f+x+AIJ9gyvMMZktv7mZEHUEru6Hq+EYroajSblWbFO3fx71i3vxFqGraEh3aUieTxss/IOwDQjGwaMBbTQa2pTRIZTE3ebX0hIc7tgpoBnwxL9PMxLIvhROVmQoXNuPbexhrHOTaaecoZ3mTNHVc4CEfx4lYLCwRvFthVKzLdRsl/9w9C3ZyiZi7s+HOi518LT1JDYjlhOJJ+jg28FssZiCufNb1Ul+Taui5LekMZi80OnQoQMbNmwotGzLli106FC1PriEEOUvISuBc0nnAGjn3c7M0ZQzVYWEC3B1P+rVcHIvhWERdwKN+u+9ShpAryqcUf2IUl0Lra4oYG2hxcZSi42u6L9azT/dvHQ24NsKarZD8QnE3sq+HA+yArB1xapxT6wa/zN4jsEACeeNxSTJV4D8H0d2np7MXD2ZOUX/zTMUHmHBmhyaai7ilJcBl/fmP/6RY+uNtlY7tH5B+YWPT0uwrBjdtsuDoigEeQfx+4XfCY8Or3KFjhCi/JS60ElLS+PcuXPG55GRkRw+fBhXV1dq1arF5MmTuXbtGt988w0Ao0eP5osvvmDSpEkMHz6c7du38+OPP7J+/fqyOwohRLW0Pzr/ak4953q4WrveoXUll5UM1w6QeymMzMhQrKIPYpWbBOTf02L5T7M41YlDhnocMtTnkFqPKNtGBPh6Uc/TngA3W2q52eHvaksNF5vCXalEyWg04F4//xH4tHGxAlj/83D5zyqqqpKYkculG+lcupHBpRsZRMancToqGX3cWVooZwlU8rsHNlIuY5kRDad+y38ABrSkuzRCUysI29pBKDWDwK1ulR4d7uZCRwgh7lapC539+/fTpUsX4/OCe2lCQkJYtmwZUVFRXL582fh67dq1Wb9+Pa+88gpz586lZs2afP311zK0tBDinoVFhwH5X4qqFH0exEVguLKftPN7Ua+G45AWiQYVHf/2LMtWLTiu1uawoR5HqU+ia0vcfevS2NeJ+3wcGeHjgJt90W7AonwpioKrnSWudpa0qlW4DMrOe4DzselERKXwS1QKF67HoIk+Qt3sU7TSnKOV5ixeShIOiScg8QQcWQpApoUTmZ6B2NVpj1VAENRoAzb/LbEqr4IrtMfjj5ORm4Gtrvpc0RJClJ1SFzqdO3dGvfUkByxbtqzYdQ4dOlTaXQkhxG0VnO2t9IVOagxcDSf3cjgZF/ZhG3cEnSETDXDz8CuXDR4cVOtzxqIhqR6B2NYMpEENdzr4OPKspz2WFnKFprKxstDSxNeRJr4FP+kmqGpnYlOzORmVwurryURduYAuaj++qccJ1JyjmRKJTV4yNtf/gut/wa78NVPta6Pxa4dtnfb59/x4NgVthRxz6I5qOtTE186X6+nXORR7iPtq3GfukIQQlVDl/AQUQlR78ZnxXEi+gIJCW++25g6n5HKzIPooXA0nMzIU9Wo4thnXgfwrNU7/NEtVbThiqMMxpQEpboHY1Qmicb26dKjhRH8HK5Qq3G2pulMUBS9Ha7wcrenS0JP8ISN6kpWr53R0Kj9diiP67H601w4QkB1BK+UstTUxOKRFQkQkRPwIQK7GmmzPltjUDkbrVzDQgY9Zj6002nm3Y935dYRFh0mhI4S4K1LoCCEqpYKrOQ1dG+Jk5XSH1maiqpAY+c+N6/vJvhSKRexxtP8MGFAw7qRBVTit1uSwoR7nrRpBjXbUqN+S1gHujPB1lHtpBADWOi0t/Zxp6ecM99cHniYqOZMDlxL56XwkmZFhuCYepSVnCdScx9GQgS46FKJD4Z+xDrJtfdD5B6EpKHx8WuYPOFEBBfkEse78OrlPRwhx16TQEUJUSgX351So0dayUvBIOY5mVwRcP4jh6n40mTeMLxfcLROnOnLYUJ8jal1uuLTEvnY7mtWpyQP+LjzlbCNXa0SJ+TjZ0LeFDX1b+AL3kZGTx9Gryay4eIPr546ivX6ARnmnCNScp6FyGauMKIhYl/8AVI0FeDVDqdkOxacVdllp+QV6BdDOK/93++SNk6TlpGFvWc1G/BNC3DMpdIQQlVLBWd6CL0PlzqCH2Ij8IYav7odr+7GIO01HVDif30RD/oABJ9UADhnqcYT65Hm3oX7DJrSr7cZoP2fsreRjWJQdW0sL2tdxo30dN+jaAIPhMS7Ep7H/YiLLzl0h+VwYdbJPGSeB9TQkQdRhiDqMBdANUC/+D2oUzOvTxmwDHfjY+1DTviZX065yMPYgD9Z8sNxjEEJUbvIXVghR6cSkx3Ap5RIaRUMb7/KcqhJIugxhi+DgCshKKvSSQv6AAYfU+hw21OWQoT4Zrk1o38CHB+p78EQdVxyszT/Rmqg+NBqFep4O1PN04KmgWhgMHTkVncrOs3GsOBPH1UtnaGo4SyvNWVppztFMuYhVZiKc25L/KFC7E7QfA/V75g+xXU6CfIK4evYq4dHhUugIIUpNCh0hRKUTHpN/NaeRayMcLR3v0LoMqCpc3gf75qOe+h1FNQCQjg2H9XU4pNbjkKEehw31yNA68mBDLzo19GJ4fXf8XGVYXFFxaDSKcZS3FzrVJTOnHWEXE9h5Jo7JZ+KIjE2isXKJVppzBGrO0UZzjlpKDET+lf9wrQNBL0CrIWDlYPJ423m3Y83ZNcauqkIIURpS6AghKp1yG1Y6LxuOr8GwbwGa6CNA/lWbXfqmLNU/zJ+GVigaLa38nHmgvgcv1HHm6tE99OsTiE4nV25ExWdjqaVTAw86NfAgNzeXlb9swMr/EfZcSOS9c/EkpOdQU4njWe1mntb+iWPCBdj4Bobt/4em9bMQNApca5ssvoLf8VMJp0jJSSmfExtCiCpDCh0hRKUTFmXigQjSYsna9xWEL8Y6+wYaIEvV8Yv+fpbpexJnW49ujT2Z38iTDnXdcbLJL2pyc3OJOmaakIQoD85W0Lt1DZ4KDsBgUDkZlcJfZ+LYcKIec68+xkDtTp7TbqRuTlT+Fc59C0gL6I59p5dQAh6AMh5Iw9PWkwDHAC6mXORA9AG61Opy55WEEOIfUugIISqVqLQorqZdRatoaeNVtvfnJJ7bT+Kfc/G79gfW5AIQrbrwTV53/rTrTXCzBkxr6k27ABcsZMhnUcVpNArNajjRrIYTY7vU43pSJptPtObd409ieWkHz2n/oJP2KA4XN8PFzcTY1COzzShqPTgUjWXZDVndzrsdF1MuEhYdJoWOEKJUpNARQlQqBX31m7o1xU5nd8/buxKfyum/VuFzahlNc49TMLbUIUM91tsNwKbFozzcvCav13CSYZ9FtebrbMOw+2oz7L7aJKS3ZWvEs7x7KJTGV75ngLITr8xzsGsSCbv+jyNej2LX8QVaNWt8z/NABXkH8dOZn2Q+HSFEqUmhI4SoVAoKnbbebe96GzfSstm0/xRZ4d/QPXUd3TRxAOSqWvZa3Uds0+EEdujOO54yb4cQxXG1s+TJtn482daPtOwB7Dx2jszQpQTH/oSPcoMuMcvJWfMtm3/pyJUGIbS7rxutaznf1cmCgt/104mnScpKwtnauYyPRghRVUmhI4SoNFRVveuBCLJy9Ww5GcPesFAaX17JQM1f2CnZoIFUjSMX/J/A66FxPFizjilCF6LKsreyoGfbRtD2I7JzZnD8r1XYH/qKgIxj9GEnnNnJ/lMNmG7zCC5tH6N/K38C3Et+Ndbdxp06TnW4kHyB/TH76ebfzYRHI4SoSqTQEUJUGtfSrhGVHoWFYkErz1Z3bG8wqOyLvMEvB66SfGIzTxnW84H2MGjzX0+0q4tFxxdxCBpCS13Z3VMgRHVlZWlFs+4h0D0E/dVDxG+djdvF9bTVnKFt9idc37WEb3b04KTPo3Rv04i+LXxxsbO843bbebfjQvIFwqPDpdARQpSYFDpCiEqj4GpOM/dm2OpuPT/NmZhU1hy8xsZDF+iYvpVR2o3U11wDLagoZAR0w+7BcbjU7lTmo0QJIfJpa7bCa9g3kBpNbujX6MOX4Jt9gzd1q8iMW8Oa9Q8w+PeHqdGgNQNb16BrI0+sddpitxXkHcQPp3+Q+XSEEKUihY4QotIo+JJT3LDSsSlZ/HrkOmsOXiMxKpKhFptZq92Osy4dAL2FHZrWz6AEv4CdW91yjVuIas3BG123d9B1eg2OryZ3z3xs4o4zxGIbQ9jG3+eas+T0w7xp2YbezWvwaKsatAtwRaP59yREwe/8uaRzJGQl4Grtaq6jEUJUIlLoCCEqBVVVjYVOkE/+/Tl5egPbTsXyfdhl/j4TSyBnedHiDx62CsdCMQBgcA5AE/wC2lZDwNrJbPELUe3prKHVEHSBg+HSnvx5eE5v4EHtMR7UHuO8wYdlB3vyXPiDuDi7MKidH4Pa+eHlaI2LtQv1XepzNvEs4dHh9Azoae6jEUJUAlLoCCEqhcupl4nNiEWn0eFt1ZDZW87wU9hFHNPO00Zzhl90O2ipufDvCgEPQPsX0TToCZriu8MIIcxAUSDgPgi4DyXxEoQtQj34DXWzo3hPs4zXLX7kx7RO/L2tBUu31SOocR2GBPvTzqudFDpCiFKRQkcIUSnsu55/Ncc9140/Zr1Me+UcIzUXsLfK+reR1gpaPAnBo8G7mZkiFUKUmIs/9HwfpfNkOPI9hC7E8cY5Rlj8wQj+AODcWV8Ona5HlIsruMPea/vMG7MQotKQQkcIUTHlZkH0UVLP7yX6xC52cRzsLeifcpoxFv/ekKxa2qPUaAN1OkProWDnbr6YhRB3x8oegkZC2+fh3FY4/jNcDYeEC9TTXKee5jo90jU84FaDy2mXCJ15HzUDOuLb9AGUmu3A0cfcRyCEqICk0BFCVAw5GXDmD7gcinp1P2r0UTSGXBwAe+C4Xw0A6qmuJDfujVP9jlCzHYp7A+maJkRVodFAgx75D4D0G3DtALmXQsk8vZv6OZc4Y2VBAhcIPnEcTiwCwOBQA41fO6jZFho8DO71zXgQQoiKQgodIYR5JV+D8K/hwFLITARA+ecRpzpy2FCfky61ibcIw1JjSeeX92CltTJryEKIcmLnBg16oGvQA+/uEBz2EWcivmWVe3tS0xSaqedoqFxGm3oNTl6Dk2th8ztQ9yFo/yLU7ZpfPAkhqiUpdIQQ5nElHPbNh5PrQNXnL1I92KJvwyFDPc7oGhEUGMjg9v4kJP8BoWEEegZKkSNENRbkE8yKiG+54ZxL77fWsu7wNd7eexqr2CMEKudor4ngQe1RNOe3wflt4FYfgl+Alk/nd48TQlQrUugIIcpPXk5+YRO6AK4dMC7eq2/CUn1Pthra0KSGM0OC/fmwpS92VvkfUV+fvvX8OUKI6qONVxs0ioZLKZdIz7vBkGB/BgfV4vCVdqwMvczoo9fxyI4iRLuZpyz+wv7GWdjwGmx/L/8evqBR4FzL3IchhCgnUugIIUwvPT6/a1rY15AWDUAOOtbmdWSZvicRBNCziTc/PVibNv6FJwJUVZX9MfuB/NnRhRDVl4OlA41dG3PixgnCosPoV7cfiqLQqpYLrWq58FbvxqwMu8yXe2oxO/VxHtf+zXCLTfhnRcOez2HvPGjUF9qPgVod8oe6FkJUWVLoCCFMJ+YE7FsAR38EfTYA8TizPLcbK/UPkWnpypNBfiy4LwB/N7tiN1EwE7q11prm7s3LM3ohRAUU5B3EiRsnCI8Op1/dfoVec7GzZGyXeox4oDa/HYni652efBPdnS6awwy32Mj9muMQ8Wv+w7tF/n08zQaChXSJFaIqkkJHCFG2DHo4syn//puLO42Lj6l1WJz7MOsN7XF1tGNEx9oMDqqFk63utpsLi87vthboGYhOe/u2Qoiqr613W5aeWGr8bCiOlYWWx9vU5LHWNdh97gZf7/LimdOtaaBcYZh2I49Z7MYq+iisHQ1bpkDb4fkPB69yPBIhhKlJoSOEKDvRx2DdOIg6DIAeDRv17Vic14uDan2a+Djx8YO16dPcF0uLko2EtD9auq0JIf7VxqsNWkXLtbRrXE+7jq+97y3bKorC/fXdub++O2diUlm8M5Jph/2ZmTWIp7V/Mky3Fc/0WPjrQ9g9F7q+nX+VR4asF6JKkEJHCHHv8rLh709Qd81CMeSRptixIrcrK/K6cx13HmrkycoHatOhjhtKKfrEG1QD4THhgAxEIITIZ6ezo6lbU47GHyU8Opz+9fqXaL0GXg589HgLXuvZkBX7LvHtPncWpffhYU04o3R/0CLvXP7Q1Cd+gf7zwLOxiY9ECGFqUugIIe7N1f2oa8eixJ9CATbog5iaO4wUC1ceC67J8PtqU8/z7oZ1PZt4luTsZGwsbGjq3rRs4xZCVFrtvNtxNP4oYdFhJS50Cng4WDGxewNe7FyXNQev8fUuJx6Ja8+T2h28Y/EdjtcOoC58AOXB1+H+V8DC0jQHIYQwubuaRWvevHkEBARgbW1NcHAwYWG37iebm5vLjBkzqFu3LtbW1rRs2ZKNGzfedcBCiAoiJwP9xrdQv+6OEn+KONWJ0TkTeFP7Gk8/1I49b3blg0eb33WRA//en9PaqzU6jdyfI4TIV9CVNTw6HFVV72ob1jotg4NrsfWVTiwZ1o5zNR6le/bHbNG3QTHkwo4PyFnYCa4fKsvQhRDlqNRXdH744QcmTpzIwoULCQ4OZs6cOfTs2ZPTp0/j6elZpP0777zDt99+y1dffUWjRo3YtGkTjz76KHv27KFVq1ZlchBCiPKVe+5vsta8iEPGFQBW6x/gM4vneLJ7Sz7u4I+jddkUJQWFjtyfI4S4WaBnIBYaC6LSo7iadhU/B7+73pZGo9C1kRddGnqy53xDPttal18vr2eabjlu8SfRL3qItNajcer1LuhsyvAohBCmVupCZ9asWYwcOZLnnnsOgIULF7J+/XqWLFnCm2++WaT9ihUrePvtt+nduzcAY8aMYevWrXz66ad8++23xe4jOzub7Oxs4/OUlBQg/+pQbm5uaUMuUwX7N3ccVZXk17TuNb/ZaUlc/fkNGl1bjQ64rrryP+1oGnd6lHXt/IwTfJbFz09v0BsHImjt3rpSvCfk/Wtakl/Tqkz51aGjmVszDscdZt+1fXjX9S6T7Qb5O/Ht8+0Iu1iXt7bdT+9rn9Ffuweng/OIPfYrWb1m49O8811tuzLltzKS/JpWRctvSeNQ1FJc883JycHW1paff/6ZAQMGGJeHhISQlJTEunXriqzj5ubGxx9/zPPPP29c9swzz7Br1y4uXrxY7H6mTZvG9OnTiyxfuXIltra2JQ1XCFFGcvSQeOUIjyQsxVtJAOBHw0Mc9B5EG29rLE0wQNG1vGssSFuAFVa85fQWWkVGQRJC/Gtr5lZ2ZO+gpa4lT9g9YZJ9RKZC8qVDjM5ZipeShEFV2KjrxrXaT+Bpb22SfQoh7iwjI4PBgweTnJyMo6PjLduV6opOfHw8er0eL6/C48x7eXlx6tSpYtfp2bMns2bN4sEHH6Ru3bps27aNNWvWoNfrb7mfyZMnM3HiROPzlJQU/Pz86NGjx20Ppjzk5uayZcsWunfvjk4n9wyUNcmvaZU2v+nZeazecwyvve/xhPo3KHAVb463nkHvbv15VGe64mNFxAo4BO1829Gvc787r1AByPvXtCS/plXZ8usW7caO7TuIsoiiV69epRrRsXR6c/LCUM78NpkH0jbSO28LV88c4ief1+na+wma+JTse0lly29lI/k1rYqW34LeXndi8lHX5s6dy8iRI2nUqBGKolC3bl2ee+45lixZcst1rKyssLIqOkuxTqerEMmFihVLVST5Na075Tc9O49ley5y8e/vmGT4Gg8lBT0aztR+lrpPfsDDNnc/wEBJHYg7AEB73/aV7r0g71/TkvyaVmXJb1uftug0OmIzY4nKisLf0d9k+2rZsA40/IFLYeux2zyRmnnRvBL9Bj8u2sDiOhMY1bMNzWo4lWhblSW/lZXk17QqSn5LGkOpRl1zd3dHq9USExNTaHlMTAze3sX3j/Xw8GDt2rWkp6dz6dIlTp06hb29PXXq1CnNroUQ5SAnz8DyPRd58eMvabB9FDPVWXgoKSQ71EMdvpnGIZ9hWQ5FTp4hjwMx+YWOzJ8jhCiOtYU1LTxaAP8OXGJq/kF9cH/9AInNh2NA4UmLv3jn0nN8P38aE7/dTWR8ernEIYQomVIVOpaWlrRp04Zt27YZlxkMBrZt20aHDh1uu661tTU1atQgLy+P1atX079/6ca9F0KYjt6g8sv+SN7/6D2ab3yM5fq36K49gEGxwPDgJJzG78GiVvkVHBE3IkjPTcfB0oGGLg3Lbb9CiMrFOMx0VHj57dTKHpfHZqMZvpEc57p4Kkm8r1vClLNPsHnuC3y4agvRyVnlF48Q4pZK3XVt4sSJhISE0LZtW4KCgpgzZw7p6enGUdiGDh1KjRo1+N///gdAaGgo165dIzAwkGvXrjFt2jQMBgOTJk0q2yMRQpSaqqr8dfg0FzbOo1fW7zyqJIAG9IoOpfnjaO4fb5bZwQvOzrb1aotWI4MQCCGK1867HQuOLCAsOgxVVU14n04xarXHcuweOLCUnD0LcE65xAva38iLWM+Wk0HENXuOR/o8irNd0a74QojyUepCZ9CgQcTFxTFlyhSio6MJDAxk48aNxgEKLl++jEbz74WirKws3nnnHS5cuIC9vT29e/dmxYoVODs7l9lBCCFK7+iBPcRsmcMDmdvprOSCAhk6N3TtR6ALHgH2RefFKi/h0flnZ2X+HCHE7bT0aImV1oobWTeITI6kjnM5d4vXWUP7MVgGjYIzm0j563Mco/bQS9kHJ/dx4uRHhDd6jvv6j0BnYf77GoSobu5qMIJx48Yxbty4Yl/bsWNHoeedOnXi5MmTd7MbIURZUw1YxBzizCezaZH9z2zfCkTbNcSx83hsWz0OFuY9+5hryOVg7EFA7s8RQtyepdaSQI9AQqNDCYsOK/9Cp4BGC41649ioN2r0caI2z8b9wjqacoGmp94l/tSnXKnzFFrrZuaJT4hqyuSjrgkhKoDsVG7sWoJh30L65F4HQK8qRDh3wvfhiXg3ehDKs8vHbZyIP0FmXibOVs7Ud6lv7nCEEBVcW++2xkLnqUZPmTscFO9m+A5djCHtI06u/wyPUyvwUBNwv7CQpqoF11K2UKv3q2h8W5o7VCGqPCl0hKjK0mJJ3z4T7eFvcTNkAJCs2hLm2o9G/V6lWZ2Kd6N/Qbe1tl5t0SilGi9FCFENBXkHMY957I/eX/736dyGxt6dJoNmkJP9Nn+vX4Lr0a9pppwj4Oo6WLSORI8gnHtMQqnf3dyhClFlSaEjRFWkqmQf+A5142Ts8vIn1Tpn8GW322NkubVg+KBHK8Q4+MUpGIhAuq0JIUqiuXtzbCxsSMxO5FzSuQp3JdjSyooHB44huecwZn61gKaJW+hOKC5xYfDd4yTVeQTnx2aDnbu5QxWiypHTpUJUMYbEy0Qv6IfV72OxzkvhhMGf95zf40bI3wx+cSreDhV3BKAcfQ6HYw8DMhCBEKJkdFodgR6BQPnNp3M3bC0taFC7Lm0n/syXrX5hqaE3elXB+cKvpM1qTXLYSlBVc4cpRJUihY4QVYXBwKWNn5H1WRDesTvJVi34UvcMVx77nXfGv0RwXQ9zR3hHx+KPkaXPwtXalbrOdc0djhCikgjy+Wc+nehynE/nLrnYWjJuQCd6TlzCnIAFRBj8sNcn47RhDJGf9yPrxhVzhyhElSGFjhBVQFTkCc7O7Iz/vnexVTM5pDbkl+AfCJn0GQ+3rFVh+qzfyc3d1ipLzEII8yvo6ro/Zj8G1WDmaErG19mGV597mqzntrHS9hlyVC21E3aS93kQR9bNRTVUjuMQoiKTQkeISiw9M5sdS6fgvKwz9TOPkKFa8avPeGpO3MFTvbthratck23K/DlCiLvRxK0Jtha2JGcncybxjLnDKZVWtb146rUv2Nl1DSeU+tiTQctDUzj2YRciIo6aOzwhKjUpdISohAwGlc1//knkx/fR+dJcbJQcjlkGcvXprTzywgw8nGzNHWKpZeuzORJ7BJCBCIQQpaPT6Gjt1RqAsKiKe5/OrWg0Cg916kztN3bzd8B4slQdLXIO47+qG2sXvENMUrq5QxSiUpJR14QopX1R+/j62NfkGfLMsv+0zBwy4i/hoY9D8VLR40OWQy3sXH3h4iy4eOt1VVUlITWBNVvXVLiuYVl5WeQYcnC3cSfAMcDc4QghKpl23u3YdW0XS08sZfuV7eYOp4gSf/56w5fOXcmLO4uNPg34gW9X/UqGQ23cXV3QmOmju6FLQ94IekOG/ReVihQ6QpTSF4e+4EjcEfMGYQlXuGn0tLwoiI0q8eoXYy+WfUxlpFPNThWuCBNCVHwP1HiAOQfmEJ8ZT3xmvLnDuaUSf/5aAlj/80SFvAtciTVNTCVxIOYAfer0oYVHC/MFIUQpSaEjRClk5GZwIv4EANM7TsdeZ2/yfebkGdh67AqWZ9bRVTmABpUsjR1qqyHY1L0PKHlRoNfrOXjwIK1bt0arrXj37+g0OoJ9gs0dhhCiEqrvUp+VfVZyPe26uUMp1t1+/qoZN0jZtQinpAgArqie7HDoR+f27fBzLZ9uyt9FfMfB2IOERYdJoSMqFSl0hCiFg7EHyVPzqGFfg4H1B5p8f9siYtjwy0+8mvUFdTX5V2yS6z6C08C7m1wuNzeXnOM5dKvVrcJOGCqEEHermXszmrk3M3cYxbqnz9/GT5Fz8HsMf7yBdd4lQjLm8+WafoS1m8D4ns1xsDbt53lcZhwHYw8SHh3OiOYjTLovIcqSFDpClMLNwx+b0tXEDD5ct5925z5jvsUW0ECWtQdW/efg1LivSfcthBCiglEULNsMhobdyFr7CtbnfmesxTrOHQhnwtFx9H/kUfq18DFZt9+Cv3mHYg+Rq89Fp5UTZaJykDvKhCiF8CjTDn+ck2dgwY7zTJ31OW9GPkeIxRYAcls+g/X4/ShS5AghRPVl74n1M9/Bk9+QY+1OPc11vsp7m7ifJvL8V39xPi7NJLut51wPFysXMvMyOX7juEn2IYQpSKEjRAml5qRyMuEkYJorOnvP3+DJuX/gtvUVFms/oKYST66DHzy7Ft2j88DGucz3KYQQohJq0h/Ll8PRN38KjaLyvMUfTL06kmlz5/PJptNk5ujLdHcaRUNb77ZA5Ry+W1RfUugIUUIHYw5iUA3UcqiFt513mW03NjWLV344zNLFn7Mo5UWetPgLFQU1aBS6cfugbpcy25cQQogqwtYV7WNfwpCfybP3xV8TywqL9/Hd+QYDZm1gW0RMme6uoCdDwcTOQlQGUugIUUIFH+5ldTVHb1D5Zu9Fnvj0Vx46/gaLLGfjqSShd62HMnwjSu+ZYGX6Ud2EEEJUYvW7YzEuFLXt8wAMtviTZZkv892KRYz8Zj9XEzPKZDcFf/sOxx0mR59TJtsUwtSk0BGihMpyIILDV5Lo/8VODvz2Jb+or9BXuw9V0cL9E9GO2Q212t/zPoQQQlQT1o4ofWfBsPUYXOrgoySwxPIT+px9l8dn/cb8HefIyTPc0y7qONXBzdqNbH02R+OOllHgQpiWFDpClEBydjKnEk4B9zYQQXJGLm//cowx83/llbgpzLWcj6uShurVDGXkdug2FXTWd96QEEII8V8B96MZsxs6voSqaBig3cPvmtc4sXk5veb8xZ7zdz+RqqIoxhN90n1NVBZS6AhRAgdiDqCiEuAYgIetR6nXV1WVX49cp+snf6Lfv4xNlpN4SHsIVWsJXd5BGbUDfAPLPG4hhBDVjKUt9Pg/lOe3ono2wV1JYZ7lZ0xK/j/Gf7WRiT8eJjH97rqeFRQ6BT0chKjoZB4dIUqg4OzV3VzNiUrO5N21xzl96hifWXzNfboT+S/UaIvSfx54NirLUIUQQgio2QZl1F+w81PUnZ/Qk/2015zk/448Q/fT3Znevzm9m3uXau6dgr+BR+KOkJWXhbWF9EAQFZtc0RGiBIz35/iU/P4cg0FlZehlHp61g1pnlrPJ8k3u055AtbCBnh/A85ulyBFCCGE6FpbQZTLKC3+DbyuclAxm6hYxK2cGH6zcxKgVB4hJySrx5vwd/fG08STXkMuRuCMmDFyIsiGFjhB3kJiVyJnEMwC08ypZoXMxPp3BX+9j8dqNLFXfYYpuBbZKNgQ8gPLiHugwFjRaU4YthBBC5PNqCs9vhW7TUS2seVB7jM1Wk/A5/Q3dZ/3JqrDLqKp6x80oimI84Sfd10RlIIWOEHdwIOYAAHWd6uJm43bbtnl6A1/+dZ4+c7bT5tIS/rCcTGvNOVRLB+g7B4b+Cq51yiFqIYQQ4iZaC7h/Asro3VCrI3ZKNjN0y/naMJUvf9nMkK9DuXQj/Y6bKTjhtz96v6kjFuKeSaEjxB2UdFjpk9dTeHT+Hn7duJEfNW/zuu5HLJU8qN8TZWwotH0ONPIrJ4QQwozc68Gw9dD7E1RLe4I0p9lo+SbNLy6j95w/+ervC+gNt766U3CfztH4o2TmZZZX1ELcFfnWJcQdGAci8Cl+IILsPD2fbj7N419s5+GYL1ln9Q5NNZdQbVxg4Fcw+AdwqlGeIQshhBC3ptFA0EiUF/dC3a5YKblM1n3P98o7rP5jEwPn7+ZUdEqxq9Z0qIm3nTd5hjwOxR4q58CFKB0pdIS4jRuZNziXdA6Atl5ti7x+4FICvefuZPefG/jVYjJjLX7FAgM0fRRlbDi0eBJKMaKNEEIIUW6ca8Eza6D/fFRrJ1poIvnN8m26Ri9m4Gd/MmvLGbLz9IVWURTFeFVH5tMRFZ0UOkLcRnhM/od4A5cGuFi7GJenZ+cx7dcTDF34J88kzudnq+nU01wHey8Y9B08sQzsSz/fjhBCCFGuFAVaDUEZGwaN+qJT9Iy3WMMvFm/x1/Y/6PvZLg5eTiy0isynIyqLuyp05s2bR0BAANbW1gQHBxMWdvs3+pw5c2jYsCE2Njb4+fnxyiuvkJVV8uEMhTCX8Kii8+fsOhtPj9l/c3bfb2zUvcFzFpvQoELgEBgbCo37mitcIYQQ4u44eMOgb+GJZah2HjTUXGWN1TSeSPiSIQv+ZMZvJ8nMyb+6U/A38UT8CdJz7zyAgRDmUupC54cffmDixIlMnTqVgwcP0rJlS3r27ElsbGyx7VeuXMmbb77J1KlTiYiIYPHixfzwww+89dZb9xy8EKZ280AEGTl5TFl3nBcXb2dc2md8Z/k//DRx4OQHz6yGAfPBxuUOWxRCCCEqKEX5p+t1GLQYhBYDoyzWs0H3Jif2bKDP5zs5fCUJX3tfatjXQK/qORhz0NxRC3FLpS50Zs2axciRI3nuuedo0qQJCxcuxNbWliVLlhTbfs+ePdx3330MHjyYgIAAevTowdNPP33Hq0BCmFtsRiwXUy6ioGCRU4/ec3dyLXQNm60m8bTFn/mN2o2EF/dCvW7mDVYIIYQoK7auMHARDP4JHGtQWxPDD1bv8Vzi54Qs2Mqszadp+88w03KfjqjILErTOCcnhwMHDjB58mTjMo1GQ7du3di7d2+x63Ts2JFvv/2WsLAwgoKCuHDhAhs2bODZZ5+95X6ys7PJzs42Pk9JyR/5Izc3l9zc3NKEXOYK9m/uOKqqipTf0GuhADhb+DNx8W7etfiG/pZ7AFBd66DvMxe1Vof8xhUg3pKoSPmtiiS/piX5NS3Jr2lVyvzW7gKjdqHZNg3toeU8a7GVruoh3t7xPJE1HcEOwqLCKsQxVcr8ViIVLb8ljUNRSzIV7j+uX79OjRo12LNnDx06dDAunzRpEn/99RehoaHFrvfZZ5/x2muvoaoqeXl5jB49mgULFtxyP9OmTWP69OlFlq9cuRJbW9uShivEPfkueS0R6n5qJtTh26QDuCmpGFA479mbUz6PYtBYmjtEIYQQoly4p54k8PIS7HLyb1VYQntm174OqsJbTm9jq7E2c4SiOsnIyGDw4MEkJyfj6Oh4y3aluqJzN3bs2MEHH3zA/PnzCQ4O5ty5c4wfP5733nuPd999t9h1Jk+ezMSJE43PU1JS8PPzo0ePHrc9mPKQm5vLli1b6N69OzqdzqyxVEUVIb96g8rXuy5yOvZTsIQ3c/bhpmShejbB0GcuAb6tCDBLZPeuIuS3KpP8mpbk17Qkv6ZV+fPbG3JfRP/X/9CEfclwdR8/5tbgmk7L8tgE5g0YTC1X852Mrvz5rdgqWn4LenvdSakKHXd3d7RaLTExMYWWx8TE4O3tXew67777Ls8++ywjRowAoHnz5qSnpzNq1CjefvttNMXMFG9lZYWVlVWR5TqdrkIkFypWLFWRufJ7MT6dV388jG/M9xjqJKJRVVrlGKDzWyj3v4KFRdW4iiPvX9OS/JqW5Ne0JL+mVanzq3OCXh9C88dR142lfWYsq3X21MlcxnPzXBjdpyNPB/mhmHH+uEqd30qgouS3pDGUajACS0tL2rRpw7Zt24zLDAYD27ZtK9SV7WYZGRlFihmtVgtAKXrNCWFSqqqyYt8lRsxdzYSoSfRwWA1AE1WH46i/oPMbUEWKHCGEEOKe1GyL8sLfBAXkD8QTb5vEr8pE9q+bx3NLw4hJkSlERMVQ6lHXJk6cyFdffcXy5cuJiIhgzJgxpKen89xzzwEwdOjQQoMV9OvXjwULFrBq1SoiIyPZsmUL7777Lv369TMWPEKYU1RyJiGL93H+t0/4VfMaD2iPE2ZjB0C7Zs+AVxMzRyiEEEJUMBZWtHvoAwBOWVqiaDOZZbmQYZGvM3TWan49ct3MAQpxF/foDBo0iLi4OKZMmUJ0dDSBgYFs3LgRLy8vAC5fvlzoCs4777yDoii88847XLt2DQ8PD/r168f7779fdkchxF1QVZV1h6+zZN0mphgW0FZ3Jn95rY6E2+dAZixBvsFmjlIIIYSomDxsPajtVJvI5Ej2Bw2lS/j3dOYIbdWJfPjj02w5/gwzBrTAxU56RAjzuKvBCMaNG8e4ceOKfW3Hjh2Fd2BhwdSpU5k6derd7EoIk0jKyOHdNYfxi/ianyzWYKXJxaCzQ9NjBtcb9uTaL73RKlpaebYyd6hCCCFEhdXOq11+oePqy0Ojd2FYNxb7q2H8n24poaf38vzssYx/shedGniYO1RRDZW665oQlV3ohRu8PPsbRp0ZySTdD1gpuRjqPoRmbCi0G0F47AEAmro3xU5nZ+ZohRBCiIqrnU/+xKFh0WHg0QDN8I3w8EcYLGwI1pxiZe5Edi2fwvu/HSM7T2/maEV1I4WOqDby9AbmbDxG+JKJLM6ZRHPNRfKsnGDAAjTPrAZnP+DfWZ6DvIPMGa4QQghR4bXzyi90ziSeITErETRaaD8azdh96Gt3wlrJ5W3dSvqED+WVz1ZyPi7NzBGL6kQKHVEtXEnI4J0vltJnzyDGWaxFp+jJa9gPi3HhEDgY/hkKU1XV/LNSQDvvduYMWQghhKjw3GzcqOdcD4D9Mfv/fcElAO3QdfDI5+TqHAjUXGBO8ng2fD6en0LPy8i7olxIoSOqvN/3n2XH3Of5IOFV6muukW3lBk8sx+Lpb8HBq1Dbq6lXiU6PxkJjIffnCCGEECVQcGIwLCqs8AuKAq2HonspnKw6PbFU9Lyk+Zlm6/vz8ZJVJGfkmiFaUZ1IoSOqrLTsPOYvXUrzX3vzrLIBjaKS3ugJrMbvh6YDil2n4GpOC/cW2FjYlGO0QgghROVU0NW7oOt3EY4+WD/7A4aBi8nUOdNYc4XXLo/h909HsP+cDEMtTEcKHVElHTt/me0zn+bFSxPw18SSaumJ/ukfsXvqa7B1veV60m1NCCGEKJ22Xm1RUDiffJ74zPjiGykKmhaPYzPhAIl1HkGrqAzRr8Xtmy78+PMP5OkN5Ru0qBak0BFVisGgsmH1Mjy+6cQjeZsBiGkwBIeJB9A27HnbdVVVlYEIhBBCiFJytnamgUsDAPZH7799Yzt3XIauIPPx70i2cKe2Jponj49iyydDuRYTWw7RiupECh1RZcTGXGPPJwPpfWw83koCsTpf0p5ai9fg+WDteMf1L6VcIi4zDp1GRwuPFuUQsRBCCFE1FPSEuGX3tf+wadYXp1cPcNH/cQB6Zf6GsqAD+zb/ZLIYRfUjhY6o/FSVo5uWYrGgPfdnbEevKpyqHYLH6/uxb9SlxJsp6LbW0qMl1hbWpopWCCGEqHKMAxJEh92h5U1snAl4bjGxA34gRuOFL/G03zOC8DlPkZ50iy5wQpSCFDqiUstKjuPknEdosXcCrqRwUVOL64//RqOQz1AsSzfZp3RbE0IIIe5OG682KChcTLlIbEbpuqB5Bj6M6+v7OegzCIOq0C7pD7LmtuXintUmilZUF1LoiErr6rnj3Jj7IE2S/yZX1fK3z3B8JoXi1/yBUm/r5vtzZCACIYQQonScrJxo5NoIKHn3tZvpbBxp/cIiTvb6gUtKDdzURGptep7w799DNchABeLuSKEjKqU9f67HbsXD1DBc5zoeHOm1hgdfmI2Vte1dbe9C8gVuZN3ASmsl9+cIIYQQd+GOw0yXQLP2PXF6ZR87HPqhUVTanf6Evz4bTmpGVlmFKaoRKXREpZKdp+eHZZ/TZkcILkoq5y3qYzFqO23bd76n7Rb0KQ70DMRSa1kGkQohhBDVS5BPfqFTqvt0iuHs6EinV74hrN4EADon/cLRT/sRcSn6XkMU1YwUOqLSuHIjne9mvc4Tke9ipeRy1uUB/F/9E0/fWve8bbk/RwghhLg3rT1bo1E0XEm9QnT6vRUlikZD0DPTudD5C7LRcZ8+jLwlvfhl50FUVS2jiEVVJ4WOqBQ2H7/Grs+fY3jGYjSKytX6Q6j/0josbBzuedsG1WAc91/uzxFCCCHujr2lPU1cmwD31n3tZnU6P0v207+QqnGkuXKBtlue5MMV60jPziuT7YuqTQodUaHl6g18tO4Ayg/P8DSbMKCQ/OA0ag6eBxptmezjXNI5ErMTsbGwoZlbszLZphBCCFEdtfO5i2Gm78Cx4QPYjfmTZJua+GniePH8GN6ds4AzMalltg9RNUmhIyqs60mZvDB/Pb0OPE937UFyFUv0jy3DqesroChltp+Cs06tPFuh0+rKbLtCCCFEdVMWAxIUR+NRD6dxf5Hq0RonJYMPM6by1bz/8fOBq2W6H1G1SKEjKqQdZ+IYN3clM+In0EITSY6lC7rh69E1H1Dm+wqLyj/rJN3WhBBCiHvT2rM1FooF19KucS3tWtlu3M4dh1EbyK7fF0tFz0zNPC6umcaknw6TmaMv232JKkEKHVGh5OkN/HZJw5LvvmWZ4R1qKvHkOtXG8oVt4Ff2AwUYVAP7Y/Lvz5GBCIQQQoh7Y6uzpal7U+DfE4llSmeD1dMrMLQfB8Brup9ofWQqT325m5jMst+dqNyk0BEVRkxKFkOXHcA+eg/LdR/iqGRgqBmEbtQ2cKtrkn2eTjhNSk4Kdjo7mrg1Mck+hBBCiOrEVN3XjDQaNA+/D70/QVU0PGWxgzcTp7LwaDa/HY0yzT5FpSSFjqgQdp2Np/ecv2l/ZTFzLOdjqeihyQA0Ib+BnZvJ9ltws2Rrz9ZYaCxMth8hhBCiuijoCh4WHWbaoaCDRqI8tRLVwoYHtcf4zuI9Pv5pB2//cozsPOnKJqTQEWamqioLdpxn+JLdTMqZx6u6nwHQtx8Hjy8FnbVJ9y/z5wghhBBlK9AzEAuNBTEZMVxJvWLanTXshfLcBlQ7DxprLrPWagqHwv7myS/3cT1J+rJVd1LoCLNJy87jxe8O8uOmP/lO938MstiBqmg4UjMEw0PTQGPat6feoOdAzAFABiIQQgghyoqNhQ0t3FsAJuy+drMarckbtolUa1+8lUR+sppB0+ureeSzv9l7/obp9y8qLCl0hFmci03j0c//olbEV/xh+SbtNGdQLe3RP7GCix4PlUsMpxJOkZabhoPOgUaujcpln0IIIUR1cHP3tXLhXIud9d/FEPAAdmTxgW4xn+dO463Fv/L1zgum7UInKiwpdES523QimknzVvJpyqtM1n2PtZILdbqgjNmDWr9nucVR8OHbxqsN2jKafFQIIYQQhQckKK8iI9fCDv3TP0PP/6Fa2NBBe5INuklEbfyEl1fuJyMnr1ziEBWH3H0timVQDWW+Tb1BZe6m41jsmc332l/RafTkWTmh6fk+BA4GRcGQm4tBNRgfplRQ6Ei3NSGEEKJstfRsiaXGkrjMOCKTIwlwCjDp/ozfHRQFQ/vR0KAHht/GY3VxF2/rvuPQ6X2M/2wCk0MeJcDdrsz3r1Hk2kFFJIWOKGJj5Ebe3vU2OYYc0+ygPiymxr/Pj32c/7jJlO+nmGbfxQjykYEIhBBCiLJkpbWipWdLwqPD6b+uf7ntt9D3BwWoXeufJ7nATPpvmGmS/bbzbsfXPb6WgqeCkZ+GKOLnsz+brsipYBq7NqaBSwNzhyGEEEJUOf3q9ENBMXcY5SI8OpxzSefMHYb4D7miIwrJ0edwOPYwACt6rcDf0f+etrd3x3pqhL5HLSUWgJTafXDsMx1sXYttn5uby9atW+nWrRs6ne6e9l0STlZOcvZFCCGEMIFH6z9Kj4Ae5OhNf/L0jt8fVJW842vJ2TwdW30KeaqGP536E/zMdBzt7e9p36//9Tqh0aGER4fLydMK5q4KnXnz5jFz5kyio6Np2bIln3/+OUFBxXf/6dy5M3/99VeR5b1792b9+vV3s3thQkfjjpKtz8bV2pWWHi1RlLs7E5OTnsSRpRPoHf8LAAkad6z6z8W/Rd/brperzcVOY4eLtUu5FDpCCCGEMB07nR12urK/J+a/SvT9od1waNyPKyvH4Xd9I08mrebSvH2kPPI5/q3ufsTX9r7tCY0OJSwqjCGNh9z1dkTZK3Wh88MPPzBx4kQWLlxIcHAwc+bMoWfPnpw+fRpPT88i7desWUNOzr+V/I0bN2jZsiVPPPHEvUUuTCI8Jn+8+3be7e66yEk6soG8dS/TzhAHwFGvATQLmYvG1rmswhSiUtHr9eTm5po7jEovNzcXCwsLsrKy0Otl1vOyoNPp0Gpl1ElRjdh74DfqBy7t/hG7LZPwV69hWPsYF44Noc6gj8Cq9Fd3CgY12h+zH4NqkJ4iFUipC51Zs2YxcuRInnvuOQAWLlzI+vXrWbJkCW+++WaR9q6uhbsorVq1CltbWyl0KqiCib0KhoUslZwM4n98CfdzPwNwVfUkpvNM2nQZUIYRClF5qKpKdHQ0SUlJ5g6lSlBVFW9vb65cuXLXJ2JEUc7Oznh7e5s7DCHKlf99T5LUqDN/LRlHp/RN1LnwLUmfbsN+8BIsAjqWaltN3Jpga2FLSk4KZxLPyNx8FUipCp2cnBwOHDjA5MmTjcs0Gg3dunVj7969JdrG4sWLeeqpp7Czu/VlzOzsbLKzs43PU1JSgPyzeeY+K1qwf3PHYQrZ+myOxB4BoJV7q9IdY1osycuexD35OAZV4RervrR89mNaeHuUajtVOb8VgeTXtP6b35iYGFJSUvDw8MDW1la+nN8jVVVJT0/Hzs5OclkGVFUlIyODuLg49Hq98cSkfD6Yhnz+mtbd5NfO0YXgl79l1ZrvuP/U/1EzJ4rcZY+Q3PszbFuV7oR8K49W7I7azd5re6nrULdU61YGFe39W9I4FLUUszhdv36dGjVqsGfPHjp06GBcPmnSJP766y9CQ0Nvu35YWBjBwcGEhobe8p4egGnTpjF9+vQiy1euXImtrW1JwxWldCH3AkvSl+CgODDJcVKJv0jYZF4n8MyneBriSFTt+dTmZVo0aISV9IYQ1ZiiKPj4+ODt7Y2Dg4O5wxHillJTU4mOjiYqKkpmjxfVVkRcFm0uL6K7Zj8A+9yfJKZmHyjhd6GdWTvZlLWJRhaNeMb+GVOGKoCMjAwGDx5McnIyjo6Ot2xXrqOuLV68mObNm9+2yAGYPHkyEydOND5PSUnBz8+PHj163PZgykNubi5btmyhe/fuVe5m+QVHF8BxuL/W/fS5r0+J1kk/8zcWP/8f9moalwye7Ggzjym9Ot/12daqnN+KQPJrWjfn12AwcPnyZVxdXbGxsTF3aFWCqqqkpqbi4OAgV3TKkE6nIzU1lQceeIC///5bPh9MRD5/Tete89sbOH29Kz9+M4En9b/TPv5HLjuAz1OfgebOX5cDbgSwadMmripX6flwT7SaqnW2t6K9fwt6e91JqQodd3d3tFotMTExhZbHxMTcsX9veno6q1atYsaMGXfcj5WVFVZWVkWW63S6CpFcqFixlJUDsQcACPINKtGxRe1agdvWCViSxxG1Hgn9vyGkTdMyiaUq5rcikfyalk6nQ6/XoygKWq0WjUZuTC0LBoMByL9aJjktO1qtFkVRsLDI/0ognw+mJfk1rXvJbzN/D3wnLmX5l+/ybPKX1Ir8kUtfXqPWqFUo1rc/0d7Usyn2OnvSctM4n3qepu5l832ooqko79+SxlCqvxSWlpa0adOGbdu2GZcZDAa2bdtWqCtbcX766Seys7N55hm5nFcRZeZlcjT+KFCCgQhUlXOrp+OzdRyW5PGXJhjrERvoUkZFjhBCCCGEObjaWTJ4/P/4vvb7ZKqW+Cfs5ursrmQlXL3tehYaC9p4tQEgLDqsPEIVJVDqU2ITJ07kq6++Yvny5URERDBmzBjS09ONo7ANHTq00GAFBRYvXsyAAQNwc3O796hFmTsce5g8Qx5etl74Ofjdsp2qz+Xkoueod2wWAL/bPUqzCWtp6OdVXqEKIYQQQpiMTqthyLCx7OiwhBuqI37ZZ0n9ojPx5w/edr2CYaYLRrAV5lfqQmfQoEF88sknTJkyhcDAQA4fPszGjRvx8sr/onv58mWioqIKrXP69Gl27drF888/XzZRizJ387DSt+r7npWWxKlZvWkS9QsGVeE33wn0eGUJbo4yQIQQIt/FixdRFIXDhw+XeJ1ly5bh7Oxs9jiEEOJmvR7ux8VH1xGJLx6GOKxX9OHc3t9u2b6gR8zB2IPkGfLKK0xxG3fVyXncuHFcunSJ7OxsQkNDCQ4ONr62Y8cOli1bVqh9w4YNUVWV7t2731OwwnQKLrMWnI34r9hrkVyf3YXG6WFkqpb81Xo2/UZNx9JC+skLURVduXKF4cOH4+vri6WlJf7+/owfP54bN27cdj0/Pz+ioqJo1qxZifc1aNAgzpw5c68hCyFEmWsT2BrdyK0c1TbFngz8N4ZwYN3nxbZt6NoQR0tH0nPTOXnjZDlHKooj31IFGbkZnIg/AUCQT9H7c04d2Yv61UPU0V/gBk6c7bWKLv2fK+8whRDl5MKFC7Rt25azZ8/y/fffc+7cORYuXGi8HzMhIaHY9XJyctBqtXh7extvbC8JGxsbPD09yyp8IYQoUzVr1KDOxC2E2nVFp+hpc+gddi2aiF5vKNROo2ho69UWkPt0KgopdET+JVY1jxr2NahhX6PQa7s3/UDNNY/ixQ2uaGqQNXQzLdo/ZKZIhai8VFUlIyfPLI/Szo0yduxYLC0t2bx5M506daJWrVr06tWLrVu3cu3aNd5++20AAgICeO+99xg6dCiOjo6MGjWq2C5jv/76K/Xr18fa2pouXbqwfPlyFEUhKSkJKNp1bdq0aQQGBrJixQoCAgJwcnLiqaeeIjU11dhm48aN3H///Tg7O+Pm5kbfvn05f/78Xf98hBDiduzt7Gg38Wf21RgGwP3XF7Nv1pOkpKcXaldwwlju06kYynUeHVExFddtTW9Q2bhiJj0ufIhO0XPaugW+L6zGwUXOugpxNzJz9TSZssks+z45oye2liX7uE9ISGDTpk28//77Reb/8fb2ZsiQIfzwww/Mnz8fwHjP5tSpU4vdXmRkJI8//jjjx49nxIgRHDp0iNdee+2OcZw/f561a9fy+++/k5iYyJNPPslHH33EpEmTgPwpCyZOnEiLFi1IS0tjypQpPProoxw+fFiGnhZCmIRGq6X9yLkcWVubpoemc1/6Fg7N6onr8B/xr+EL/Ptd6lDsIXL1uei05h+KuTqTQkcQHpV/1qHglzM1M4cdCyfQL/k7UOCke08ajfoGjaW1OcMUQpSDs2fPoqoqjRs3Lvb1xo0bk5iYSFxcHABdu3bl1VdfNb5+8eLFQu2//PJLGjZsyMyZM4H8ezaPHz/O+++/f9s4DAYDy5Ytw8HBAYBnn32W7du3Gwudxx57rFD7JUuW4OHhwcmTJ0t1f5AQQpRWywETiPSqjeemUbTSH+PcV90J6/8dQa0CqedcD2crZ5Kykzh+4zitPFuZO9xqTQqdai4tJ42TCfk3zAV5B3ElLonTi0Lol7sDgNMNXqDJ0x+BzEIuxD2x0Wk5OaOn2fZdWiXt7ta2bdvbvn769GnatSs8yElQ0B3m6iK/W1xBkQPg4+NDbGys8fnZs2eZMmUKoaGhxMfHGycTvXz5shQ6QgiTq92hPwmeNcj67knqGa4Su7Y/G2IX0rtnL9p5t2PLpS2ER4dLoWNmcn2/mjsYexCDasDPwY/Y61nEzO9Nt9wd5KHhyv0f0nDwx1LkCFEGFEXB1tLCLI9bDRlfnHr16qEoChEREcW+HhERgYuLCx4eHgDY2dmVSX7+67+zXiuKYixmAPr160dCQgJfffUVoaGhhIaGAvkDIgghRHlwrdsWu7F/ct2qDp5KEp33hLDq20W08co/uSMDEpifFDrVXFhU/i9hTaU2tt/2pq16ggxsSH70O/y6jTFzdEKI8ubm5kb37t2ZP38+mZmZhV6Ljo7mu+++Y9CgQSUunho2bMj+/fsLLQsPv7ebdG/cuMHp06d55513eOihh4zd6YQQorxZu/njM+FPLjsHY6tk88TZScRtzz/xcjj2MDl6OfliTlLoVHMFZxu6nt1IPeUaCVp3GL4Rt5a9zRyZEMJcvvjiC7Kzs+nZsyd///03V65cYePGjXTv3p0aNWrc8f6am73wwgucOnWKN954gzNnzvDjjz8a51orzZWmm7m4uODm5saiRYs4d+4c27dvZ+LEiXe1LSGEuFeKjTO1XlrPZf+BaBWV8XFLsdNbkK3P5mjcUXOHV61JoVONxacnEXEjv3vKQ9kJRNvUw+mlv7CtFWjewIQQZlW/fn32799PnTp1ePLJJ6lbty6jRo2iS5cu7N27F1dX1xJvq3bt2vz888+sWbOGFi1asGDBAuPw1FZWVncVn0ajYdWqVRw4cIBmzZrxyiuvGAc7EEIIs9DqqDVsCddavYoCPJCZDMD603+aN65qTgYjqKaSMnKYvfQVcICAnFxynYPxHfkjWDuaOzQhRAXg7+9vvPJyK/8dYQ3yBxH470AGjzzyCI888ojx+fvvv0/NmjWxts4fyXHYsGEMGzbM+Pq0adOYNm1aoW1MmDCBl19+mZSUFAC6devGyZOFZx6/eb/FxSGEECalKNToP4UbHgG03vs2G+3heMQK/nLsS6dWxY9kKUxLruhUQxfjUtk8eySOhu0ANLGsge+Lv0mRI4Qwifnz5xMeHs6FCxdYsWIFM2fOJCQkxNxhCSGESbh1HEqLh/KvMp+3NuC1tj8/bv5LTr6YgRQ61Uz42eucnfcYT+auJfyfs6ldHngNZEIrIYSJnD17lv79+9OkSRPee+89Xn311SJXbIQQoipp0nIQ7lYu5CoKKTZJdNs9mEUrvydPb7jzyqLMSKFTjazfdxTtikfoTihxGh2nrSwBaOvT7g5rCiHE3Zs9ezbXr18nKyuLM2fO8O6772JhIT2nhRBVl6IoBPl2AGCHU01clTSGnXmZBQs+JTUr18zRVR9S6FQDqqqyZN0Wmm54nNaas2Ro7NnfYwoAdZ3q4m7jbuYIhRBCCCGqliDv/MmRT/g3IdanC1ZKLi/F/x+r5rzO1YR0M0dXPUihU8Vl5eqZs2QFAw4OI0ATQ5KVD9YvbOOQkgZAO2+5miOEEEIIUdYKCp2jN07g8Ny3xDcZBsDIrKXs+fw5Dl2MM2N01YMUOlXYjbRsvvjiE168PBFXJY0Ep6Y4v/Q3Gq9GhEfnT9gX5BNk5iiFEEIIIaqemg418bbzJs+Qx6H4o7g/MYfkB6dhQOFJdROJS55k06Hz5g6zSpNCp4q6EJvKD3Nf57XkD7BSckmo2Q3XsVvA3pMbmTc4l3QOgLZebc0cqRBCCCFE1aMoivGqTnh0OCgKTl1fIfvRJeQolnTVHMTnl8dYsXmfjMhmIlLoVEEHIuM4MP85XsxdDkBS8+G4Dv8RLO0ACI/Jv5rTwKUBLtYuZotTCCGEEKIqK7hFICw6zLjMpuVAtMN+J93CmRaaSLrsHsK8H35Db5Bip6xJoVPFbDl8npSlT/AEWzCgkNZ5Bs6PzQaN1tgmPCq/0JH7c4QQQgghTKfgu9aJ+BOk5/47AIHWPxi7MdtJtqlFTSWeoREvMOerr8nM0Zsr1CpJCp0q5Mc/w/FeM5AumkPkKJbkDlyKfefxRdoVXNGRQkcIUR117tyZCRMmmHQfiqKwdu3aMtnWjh07UBSFpKSkW7ZZtmwZzs7OZbI/IUTZqWFfgxr2NdCreg7FHir8oltdnMbtIMGtNY5KBi9df4MvP3+fhPQc8wRbBUmhUwUYDCpf/byejjueornmIulaZzTDfseqxaNF2sZlxBGZHImCIvfnCCGqtFsVCGvWrOG9994zT1DFOHPmDP3798fd3R1HR0fuv/9+/vzzz1JtY9CgQZw5c8b4fNq0aQQGBpZxpEKIu1Fc9zUjOzdcR//BDf/eWCp6JqR+ytq547kUn1bOUVZNUuhUctl5euYtWcKgYyOoqcSTaOOP7Yt/YuEfXGz7gtHWGrk2wsnKqTxDFUKICsHV1RUHBwdzh2HUt29f8vLy2L59OwcOHKBly5b07duX6OjoEm/DxsYGT09PE0YphLhbxgEJ/rl1oAidNW4h35HYagwAw3NWcnjeMxy+JMNP3yspdCqx5Ixcvv78fUZfeR1HJYN419a4vLQDxa3OLdcpOJsg3daEKGeqCjnp5nmUYjSfgIAA5syZU2hZYGAg06ZNA/K7ZH399dc8+uij2NraUr9+fX799ddC7Tds2ECDBg2wsbGhS5cuLFu2rNCVleKuNsyZM4eAgADj8/DwcLp37467uztOTk506tSJgwcPFlrndrFcvHiRLl26AODi4oKiKAwbNgwo3HWt4KrPfx8FbQHWrVtH69atsba2pk6dOkyfPp28vDzj62fPnuXBBx/E2tqaJk2asGXLlhJmG+Lj4zl79ixvvvkmLVq0oH79+nz44YdkZGRw/PjxQm13795NixYtsLa2pn379oVev7nr2rJly5g+fTpHjhwxHs+yZctKHJMQomwVfOc6mXCS1JzU4htpNLj0/5CUhz5Gj4b+6p+kL3mUPw+fK8dIqx4Lcwcg7s7VhHS2ffkqY7O/BwXi/Pvg8cwS0Fnfdj3j/DneMn+OEOUqNwM+8DXPvt+6bhx1sSxMnz6djz/+mJkzZ/L5558zZMgQLl26hKurK1euXGHgwIGMHTuWUaNGsX//fl599dVS7yM1NZWQkBA+//xzVFXl008/pW/fvoSHh+Po6HjHWPz8/Fi9ejWPPfYYp0+fxtHRERsbmyL76dixI1FRUcbnERER9O7dmwcffBCAnTt3MnToUD777DMeeOABzp8/z6hRowCYOnUqBoOBgQMH4uXlRWhoKMnJyaW6/8fNzY2GDRvyzTff0Lp1a6ysrPjyyy/x9PSkTZs2hdq+/vrrzJ07F29vb9566y369evHmTNn0Ol0hdoNGjSI48ePs3HjRrZu3QqAk5NcwRfCXLztvKnlUIvLqZc5GHOQTn6dbtnW8YEXyHSthfLzMO7jGKfXDGBN4mIGdim+p464PbmiUwmdvBLHoc+HEJL9PQDxgWPxCPn2jkVOdHo0l1Mvo1E0tPZqXR6hCiGqoGHDhvH0009Tr149PvjgA9LS0ggLy79avGDBAurWrcunn35Kw4YNGTJkSKGrIyXVtWtXnnnmGRo1akTjxo1ZtGgRGRkZ7N69u0SxaLVaXF1dAfD09MTb27vYL/uWlpZ4e3vj7e2NTqdjxIgRDB8+nOHDhwP5hdSbb75JSEgIderUoXv37rz33nt8+eWXAGzdupVTp07xzTff0LJlSx588EE++OCDEh+noihs3bqVQ4cO4eDggLW1NbNmzWLjxo24uBQe/n/q1Kl0796d5s2bs3z5cmJiYvjll1+KbNPGxgZ7e3ssLCyMx1ZckSeEKD+3vU/nP2ya9kL7/EZSLNxoqLnCfTueZNnqdTLXzl2QKzqVzO7jF9D89Cz9lOPo0ZD60Ee4PzCqROsWXM1p7NoYB8uK0z9diGpBZ5t/ZcVc+y5DLVq0MP7fzs4OR0dHYmNjgfwrIsHBhc88dujQodT7iImJ4Z133mHHjh3Exsai1+vJyMjg6tWrJY6lNHJzc3nsscfw9/dn7ty5xuVHjhxh9+7dvP/++8Zler2erKwsMjIyiIiIwM/PD1/ff6/WleZ4VVVl7NixeHp6snPnTmxsbPj666/p168f4eHh+Pj4FLtdV1dXGjZsSERERKmPVQhR/tp5t2P12dXG72J3oqvZCotxO4hf1B+vjAs8fnQUi5KieC5kFJYWcp2ipKTQqUR+3xlGgy3P0UBzlSzFGv1jy3Bu1qvE60u3NSHMSFHKtPuYqWg0miJnDXNzcws9/29XKUVRMBgMZbqPkJAQbty4wdy5c/H398fKyooOHTqUeSwFxowZw5UrVwgLC8PC4t8/jWlpaUyfPp2BAwcWWcfa+vZX0Uti+/bt/P777yQmJhq75M2fP58tW7awfPly3nzzzXvehxDC/Aq+e51KOEVydnKJBoRSnGvh/vIOYr56Eq8b+3j+8mSWfXGdJ0e/i6O17o7rC+m6Vimoqsr3634jaOsTNNBcJcnCHc3zG7ErRZEDMhCBEOLOPDw8Ct2zkpKSQmRkZInXb9y4sbEbW4F9+/YV2Ud0dHShYufw4cOF2uzevZuXX36Z3r1707RpU6ysrIiPjy/FkeR3S4P8KzC3M2vWLH788UfWrVuHm5tboddat27N6dOnqVevXpGHRqOhcePGXLlypVDO/nu8t5ORkQHkF38302g0RQq2m7ebmJjImTNnaNy4cbHbtbS0vONxCyHKj4etBwGOAaioHIg5UPIVrZ3wGvMbUbUHYqEYGJE0lz9mv0BUUvqd1xVS6FR0eXoDy5Yt4pGDz+OpJBFnUxfHsTuwrNmqVNu5nnada2nX0CpauT9HCHFLXbt2ZcWKFezcuZNjx44REhKCVqst8fqjR4/m7NmzvP7665w+fZqVK1cWGfGrc+fOxMXF8fHHH3P+/HnmzZvHH3/8UahN/fr1WbFiBREREYSGhjJkyJBS32fi7++Poij8/vvvxMXFkZZWdF6KrVu3MmnSJGbOnIm7uzvR0dFER0eTnJwMwJQpU/jmm2+YPn06J06cICIiglWrVvHOO+8A0K1bNxo0aEBISAhHjhxh586dvP322yWOsUOHDri4uBjXP3PmDK+//jqRkZH06dOnUNsZM2awbds2jh8/zrBhw3B3d2fAgAHFbjcgIIDIyEgOHz5MfHw82dnZJY5JCGEaxmGmS9h9zcjCEp+hS4hpMxGAQdmrOfbZk5y+WvpuutXNXRU68+bNIyAgAGtra4KDg4ucvfuvpKQkxo4di4+PD1ZWVjRo0IANGzbcVcDVSWaOnpXzpzP04hvYKdlEubXHY/yfaFz8Sr2tgqs5Td2bYqer+N1nhBDmMXnyZDp16kTfvn3p06cPAwYMoG7duiVev1atWqxevZq1a9fSsmVLFi5cWOTm/MaNGzN//nzmzZtHy5YtCQsL47XXXivUZvHixSQmJtK6dWueffZZXn755VLPE1OjRg3jYAJeXl6MGzeuSJtdu3ah1+sZPXo0Pj4+xsf48eMB6NmzJ7///jubN2+mXbt2tG/fntmzZ+Pv7w/kX3n55ZdfyMzMJCgoiBEjRhS6n+dO3N3d2bhxI2lpaXTt2pW2bduya9cu1q1bR8uWLQu1/fDDDxk/fjxt2rQhOjqa3377zXjV6r8ee+wxHn74Ybp06YKHhwfff/99iWMSQphGO5+SD0hQhKLg1W8qN7rPJQ8tPQy7SPu6HwcjzpdxlFWLopZyCIcffviBoUOHsnDhQoKDg5kzZw4//fQTp0+fLvaPUE5ODvfddx+enp689dZb1KhRg0uXLuHs7FzkQ/xWUlJScHJyIjk5udCwouaQm5vLhg0b6N27d5G+4WUpOT2bP+e9yICMnwG4FjCQGs8uAu3d7fPtXW/z6/lfGdF8BONbjy/LUMtUeeW3upL8mtbN+dXr9URGRlK7du0yuZejMtuxYwddunQhMTHRONfL3TAYDKSkpODo6Fikq5e4e1lZWURGRlKzZk22b98unw8mIp+/plUZ8nsj8wadf+wMwN+D/sbF2uX2K9xCWsQ2lB+fxU5NJ1L14WrvFTwQbNrbEipafktaG5R6MIJZs2YxcuRInnvuOQAWLlzI+vXrWbJkSbE3TS5ZsoSEhAT27NljTMzNk8IVJzs7u9Bl9pSUFCA/yf+9EbW8FezflHFEJyRx/qsQBuTlD6N6ucV4fPq+Q64BMJR+v6qqEhaVf/agtXtrs+fwdsojv9WZ5Ne0bs6vXq9HVVUMBsNd3RxflRQc/73mouC8XEFeRdkwGAyoqmqcBFU+H0xDPn9NqzLk19HCkbpOdTmffJ7Qa6E8VOuhu9qOVb0HyR62gfhvnqC2PgqHDQPZlPgFXR/qXcYR/6ui5bekcZTqik5OTg62trb8/PPPhfoFh4SEkJSUxLp164qs07t3b1xdXbG1tWXdunV4eHgwePBg3njjjVv2+542bRrTp08vsnzlypXY2pbtMKkVTVJaKi3OzKWVcoZcVctfPiNI97nvnrZ5Q3+D2amz0aLlbae3sVSK7+oghCg7BXOY+Pn53bJ7UXWxa9cu+vXrx8WLF6vdxJWffvops2fPLva19u3b8/PPP5dzREXl5ORw5coVoqOjjcWOEMI0fs/4nX05+wi2DKafbb972pYuJ4lGEbOoY7hIlqrjG+cX8azdBkUpo2ArsIyMDAYPHly2V3Ti4+PR6/V4eXkVWu7l5cWpU6eKXefChQts376dIUOGsGHDBs6dO8eLL75Ibm4uU6dOLXadyZMnM3HiROPzlJQU/Pz86NGjR4XourZlyxa6d+9e5pfuTp08iuMvQ/BXokjDltQBS+nU7O6q/ZutPb8WQqG5R3MGdB9wz9szJVPmV0h+Te3m/Or1eq5cuYK9vX2177pW0JXvXqmqSmpqKg4ODiiV5C/5+PHjefbZZ4t9zcbGxux/0yC/65qNjQ0dO3bk77//ls8HE5HPX9OqLPm1umzFvl37iLeJp3fve78Co/bqx9mvhlA/eQ8jkj5jU9xLdB36LhpN2X5GVrT8FvT2uhOTz6NjMBjw9PRk0aJFaLVa2rRpw7Vr15g5c+YtCx0rKyusrKyKLNfpdBUiuVD2sRzasxn/TcNxVVKJ0XhiNXQ1PgEt7rxiCRyIyx/GMMgnqMLk704q0s+6KpL8mpZOp0Oj0aAoChqNRu4nKSMF3dUK8loZuLu74+7ubu4wbqvgvVowf5B8PpiW5Ne0Knp+g2vkT6h8Pvk8KXkpuNm43WGNO9C5Uv/l3zi5ZDRNrv1Er2ufsWPBNTq++CWWlmWfh4qS35LGUKq/FO7u7mi1WmJiYgotj4mJwdvbu9h1fHx8aNCgQaFuao0bNyY6OpqcnJzS7L7KCtuwjMabBuOqpHJBVx/7sTtwLqMiR1VVwqNkolAhhBBCCHNzsXahgUsDAMJjSjnM9K1oLWgy4iuON3kVgM5Jqzk2ux/paSW76lGVlarQsbS0pE2bNmzbts24zGAwsG3bNjp06FDsOvfddx/nzp0rdOPomTNn8PHxqfb91lFVQr+bQdvQCVgruRyz60DNV7Zj51ajzHZxKeUSsZmx6DQ6WnqUbJQ7IYQQQghhGsb5dKLKqNABUBSaPTmFE/d9Rraqo03mXq7NeYiEmCtlt49KqNTX/idOnMhXX33F8uXLiYiIYMyYMaSnpxtHYRs6dCiTJ082th8zZgwJCQmMHz+eM2fOsH79ej744APGjh1bdkdRCan6PA4sHEnw2U/RKCphHo/R9JXfsbQt2/7aBWO1t/RoibVF9b5PQAghhBDC3Np538N8OnfQtHsIl/p+TyIONMg7Q/bCrkSfP1zm+6ksSn2PzqBBg4iLi2PKlClER0cTGBjIxo0bjQMUXL58uVDfaT8/PzZt2sQrr7xCixYtqFGjBuPHj+eNN94ou6OoZPIyUzk9/ynapO4CYFedCdz3zFQUE/Q5L5h9V7qtCSGEEEKYXxuvNigoXEy5SGxGLJ62pZsM+U4atOvOJacNpK98gppqNCkrenP5kWXUat2jTPdTGdzVYATjxo0rdoZpyJ8Y7r86dOjAvn377mZXVU5W4nWiFw6gafZpslUd4a0/5P7+I0yyL1VVjYVOwdkDIYQQQghhPk5WTjRybUREQgTh0eH0qdOnzPfh36AFMS9s5cTXj9NUfwrrX5/mfNIn1O36XJnvqyKrHMPWVBGpV06Q8kVnArJPk6g6cLjrNyYrcgAikyO5kXUDK60VLTzKZnADIYSo7Dp37syECRNMug9FUVi7dm2ZbGvZsmU4Ozvfts20adMIDAwsk/0JIUyv4AR0wQlpU/Dy8aPGy5vZa3UfluRR9+8JnPt5GpR8Cs1KTwqdcpJwYhss6Y6nPoZLeHPp0XUEdzLdDLbwb9/PQI9ALLXVfOAHIUS1s2PHDhRFISkpqdDyNWvW8N5775knqNvIzs4mMDAQRVE4fPhwqdZ97bXXCg0UNGzYsEITewshKhbjgAQmLHQAnJ2cCJy4jj8cnwCg3vHZXFjyPOhzTbrfisLk8+hUJTn6HMKjwzmfe56w6DDjnAN3knx8M3YH5mNhpee8Ug+r7lOo6ZVHaFSoSePddjn/j550WxNCiH+5urqaO4RiTZo0CV9fX44cOVLqde3t7bG3tzdBVEIIU2jt1RqNouFy6mW2XdqGvaVpf38dBz3PzPWW3H9jNZq43zmxsAeefaeApW2J1s/Ly+N87nkup16mrmtdk8ZalqTQKYWUnBRGbx8NwNLtS0u3sk/BhHE5cOwdOFa2sd1OkI8MRCCEuamqSmZepln2bWNhg6KUbJbsgIAAJkyYUKhrV2BgIAMGDGDatGkoisJXX33F+vXr2bRpEzVq1ODTTz/lkUceMbbfsGEDEyZM4MqVK7Rv356QkBCee+45EhMTcXZ2Ztq0aaxdu7bQVYs5c+YwZ84cLl68CEB4eDhvvfUWhw4dIjc3l8DAQD799FPq1atnXOd2sVy8eJEuXboA4OLiAkBISAjLli2jc+fOBAYGMmfOHHbs2GFsd7OCtgDr1q1j+vTpnDx5El9fX0JCQnj77beNJ7vOnj3L888/T1hYGHXq1GHu3LklyvXN/vjjDzZv3szq1av5448/im2zdu1aXn/9da5cuUKnTp34+uuv8fPzAyiU02nTprF8+XJjjgD+/PNPOnfuXOq4hBCm4WDpQBPXJhy/cZwJOyaUz06t4BvfgoEP4mHHy6XehOGcgdeCXivbuExICp1S0Cpa6jnVIzU1FQcHB7jD9wZ90jW0OckAJCtO2Lv5otWU7MtGWWnk2ogW7nJ/jhDmlpmXSfDKYLPsO3RwKLa6kp21K4np06fz8ccfM3PmTD7//HOGDBnCpUuXcHV15cqVKwwcOJCxY8cyatQo9u/fz6uvvlrqfaSmphISEsLnn3+Oqqp8+umn9O3bl/DwcBwd/x2G/1ax+Pn5sXr1ah577DFOnz6No6MjNjY2RfbTsWNHoqKijM8jIiLo3bs3Dz74IAA7d+5k6NChfPbZZzzwwAOcP3+eUaNGATB16lQMBgMDBw7Ey8uL0NBQkpOTS33/T0xMDCNHjmTt2rXY2hb/c8rIyOD999/nm2++wdLSkhdffJGnnnqK3bt3F2n72muvERERQUpKCkuX5p+Uq6hXsYSozka1GMWCIwvINZRvN7LklFTss66jRY9BsUDj4g8WVrdfSc3/XHazcSufIMuIFDql4GLtwo99fmTDhg307t0bnU5XfMOsFBKXD8Yl6jR6VeFrhzEMenE6zrZyn4wQovIbNmwYTz/9NAAffPABn332GWFhYTz88MMsWLCAunXr8umnnwLQsGFDjh07xkcffVSqfXTt2rXQ80WLFuHs7Mzu3bt58sknSxRLwZd7T0/PW97Mb2lpibe3NwA3btxgxIgRDB8+nOHDhwP5hdSbb75JSEgIAHXq1OG9995j0qRJTJ06la1bt3Lq1Ck2bdqEr6+vMY5evXqV6DhVVWXYsGGMHj2atm3bGq9o/Vdubi5ffPEFwcH5xfLy5ctp3LgxYWFhBAUVvmpvb2+PjY0N2dnZxmMTQlQ8XWp1oUutoleUy8Oa7btpvmMk9TVXybqegO6pFWgbdLtl+9zc3Pzvv41Me395WZNCp6wlXyN1yQBcks+QoVox3/0txowah52VpFqI6szGwobQwaa9L+92+y5LLVr8e5XYzs4OR0dHYmNjgfwrIgVfxgt06NCh1PuIiYnhnXfeYceOHcTGxqLX68nIyODq1asljqU0cnNzeeyxx/D39y/U9ezIkSPs3r2b999/37hMr9eTlZVFRkYGERER+Pn5GYscKN3xfv7556SmphaaaLs4FhYWtGv37/2WjRo1wtnZmYiIiCKFjhBC3MnArvfxu91P3Fg/gvacRL/yCXL7zEbXbpi5QytT8u27LEUdJXP5YzhkxRKrOrOo5v94bdggrHVac0cmhDAzRVHKtPuYqWg0GtT/DD2am1u4W8V/r2YrioLBYCjTfYSEhHDjxg3mzp2Lv78/VlZWdOjQocxjKTBmzBiuXLlCWFjhgWbS0tKYPn06AwcOLLKOtbV1qffzX9u3b2fv3r1YWRXuNtK2bVuGDBlivNdGCCHKWt/gpmy1/ZG1P41hgGYn2vXjyUm8hGX3KVDC+zorOil0ysq5reR+/yw2+gzOGGrwQ4NZTH66BxZaGcFbCFF5eHh4FLpnJSUlhcjIyBKv37hxY3799ddCy/47YbSHhwfR0dGoqmq8Wf6/wynv3r2b+fPn07t3fjeJK1euEB8fX5pDwdIyv7uwXq+/bbtZs2bx448/smfPHtzcCvc/b926NadPny40CMLNGjduzJUrV4iKisLHxwcoery389lnn/F///d/xufXr1+nZ8+e/PDDD4WujOXl5bF//37j1ZvTp0+TlJRE48aNi92upaXlHY9bCCG6Nfdjj81SFqx4gzHKaiz3zMovdh5bcOf7dioB+RZeBtQDyzF8+yQ6fQZ79E1YE7iEtwf3lCJHCFHpdO3alRUrVrBz506OHTtGSEgIWm3Jr0qPHj2as2fP8vrrr3P69GlWrlxpHL2sQOfOnYmLi+Pjjz/m/PnzzJs3r8hIY/Xr12fFihVEREQQGhrKkCFDih1M4Hb8/f1RFIXff/+duLg40tLSirTZunUrkyZNYubMmbi7uxMdHU10dDTJyfkDyUyZMoVvvvmG6dOnc+LECSIiIli1ahXvvPMOAN26daNBgwaEhIRw5MgRdu7cydtvv13iGGvVqkWzZs2MjwYNGgBQt25datasaWyn0+l46aWXCA0N5cCBAwwbNoz27dvfsttaQEAAR48e5fTp08THxxe5EiaEEAU61vOgw4hZTFXGkKtqsYxYTc6yAZCZaO7Q7pl8E78Xqoq6dQbKby+jQc9q/f2E3f81bzwajKacR1cTQoiyMHnyZDp16kTfvn3p06cPAwYMoG7dks+ZUKtWLVavXs3atWtp2bIlCxcu5IMPPijUpnHjxsyfP5958+bRsmVLwsLCeO21wsOVLl68mMTERFq3bs2zzz7Lyy+/jKenJ6VRo0YN42ACXl5ejBs3rkibXbt2odfrGT16ND4+PsbH+PHjAejZsye///47mzdvpl27drRv357Zs2fj7+8P5HfD++WXX8jMzCQoKIgRI0YUup+nrNja2vLGG28wePBg7rvvPuzt7fnhhx9u2X7kyJE0bNiQtm3b4uHhUezobEIIUSDQz5kho9/mFYu3SVVtsLy6h9xF3SDxorlDuyeK+t+O0hVQSkoKTk5OJCcnFxpW1ByMo070eAjt+glojv8EwJy8gdj3eIcRD1aeSZQqImN+bzeqnbhrkl/Tujm/er2eyMhIateuXSb3clRmBXPVFMyjc7cMBgMpKSk4Ojqi0ch5urKSlZVFZGQkNWvWZPv27fL5YCLy+Wtakt+ycflGBu9+9QP/y5yBr5JAno07Fs/8SK5niwqV35LWBvKX4i7o8tLQrHwczfGfyFW1vJ77Ar79Z0iRI4QQQgghKq1abrZ8/OLTTHT8lBMGfywy4zEs6Y1yeoO5Q7srUuiUVuJF7j/zHtore0lRbRihf4MuT73Ck+38zB2ZEEKICuaDDz7A3t6+2EdJ59oRQojy5OVozYLRfXnP/RN26Fui0Weh/TmE2nGbzR1aqcmoa6URdRTtikdxzI7nmurGGMMbvDp0IJ0aeJg7MiGEqLA6d+5cZDjp6mL06NGFJji9WWkHVxBCiPLiYmfJ1y905YVldly/8gmDLbbT4uq36Lc5w8Nlfx+iqUihUwo3NC7kZmm4YfDnJc1kPn6+J20DXM0dlhBCiArK1dUVV1f5OyGEqHzsrSxYPLw9476bzJWznryhW8WRdFdamzuwUpBCpxSi8hx5Vf8OiQZbvhrRlZa15I+XEOL27mbySiHKU8F7VKkiEwQKIcqOtU7Lgmfb8uoPFvQ61pLWyv1S6FRVzWo4MfWZXpw4sJcmPuYd/U0IUbFZWlqi0Wi4fv06Hh4eWFpayhfJe2QwGMjJySErK0tGXSsDqqqSk5NDXFwcGo2mQoykJISoeHRaDTMfa8aM9KtM6dPI3OGUihQ6pdQuwIW4k+aOQghR0Wk0GmrXrk1UVBTXr183dzhVgqqqZGZmYmNjI0VjGbK1taVWrVqSUyHELWk0CkGeaqWbJ1IKHSGEMBFLS0tq1apFXl4eer3e3OFUerm5ufz99988+OCDcvWhjGi1WiwsLFAUhdzcXHOHI4QQZUoKHSGEMCFFUdDpdPLFvAxotVry8vKwtraWfAohhLgj6eQshBBCCCGEqHKk0BFCCCGEEEJUOVLoCCGEEEIIIaqcSnGPTsGM2ikpKWaOJP9m2IyMDFJSUqSPuAlIfk1L8mtakl/TkvyaluTXtCS/piX5Na2Klt+CmqCgRriVSlHopKamAuDn52fmSIQQQgghhBAVQWpqKk5OTrd8XVHvVApVAAaDgevXr+Pg4GD2cf5TUlLw8/PjypUrODrKpKFlTfJrWpJf05L8mpbk17Qkv6Yl+TUtya9pVbT8qqpKamoqvr6+t51AulJc0dFoNNSsWdPcYRTi6OhYIX7QVZXk17Qkv6Yl+TUtya9pSX5NS/JrWpJf06pI+b3dlZwCMhiBEEIIIYQQosqRQkcIIYQQQghR5UihU0pWVlZMnToVKysrc4dSJUl+TUvya1qSX9OS/JqW5Ne0JL+mJfk1rcqa30oxGIEQQgghhBBClIZc0RFCCCGEEEJUOVLoCCGEEEIIIaocKXSEEEIIIYQQVY4UOkIIIYQQQogqRwodIYQQQgghRJUjhc5/vP/++3Ts2BFbW1ucnZ1LtI6qqkyZMgUfHx9sbGzo1q0bZ8+eLdQmISGBIUOG4OjoiLOzM88//zxpaWkmOIKKrbR5uHjxIoqiFPv46aefjO2Ke33VqlXlcUgVyt28zzp37lwkd6NHjy7U5vLly/Tp0wdbW1s8PT15/fXXycvLM+WhVEilzW9CQgIvvfQSDRs2xMbGhlq1avHyyy+TnJxcqF11fv/OmzePgIAArK2tCQ4OJiws7Lbtf/rpJxo1aoS1tTXNmzdnw4YNhV4vyedxdVKa/H711Vc88MADuLi44OLiQrdu3Yq0HzZsWJH36sMPP2zqw6iwSpPfZcuWFcmdtbV1oTby/i2sNPkt7m+Zoij06dPH2Ebev//6+++/6devH76+viiKwtq1a++4zo4dO2jdujVWVlbUq1ePZcuWFWlT2s90k1NFIVOmTFFnzZqlTpw4UXVycirROh9++KHq5OSkrl27Vj1y5Ij6yCOPqLVr11YzMzONbR5++GG1ZcuW6r59+9SdO3eq9erVU59++mkTHUXFVdo85OXlqVFRUYUe06dPV+3t7dXU1FRjO0BdunRpoXY357+6uJv3WadOndSRI0cWyl1ycrLx9by8PLVZs2Zqt27d1EOHDqkbNmxQ3d3d1cmTJ5v6cCqc0ub32LFj6sCBA9Vff/1VPXfunLpt2za1fv366mOPPVaoXXV9/65atUq1tLRUlyxZop44cUIdOXKk6uzsrMbExBTbfvfu3apWq1U//vhj9eTJk+o777yj6nQ69dixY8Y2Jfk8ri5Km9/Bgwer8+bNUw8dOqRGRESow4YNU52cnNSrV68a24SEhKgPP/xwofdqQkJCeR1ShVLa/C5dulR1dHQslLvo6OhCbeT9+6/S5vfGjRuFcnv8+HFVq9WqS5cuNbaR9++/NmzYoL799tvqmjVrVED95Zdfbtv+woULqq2trTpx4kT15MmT6ueff65qtVp148aNxjal/ZmVByl0bmHp0qUlKnQMBoPq7e2tzpw507gsKSlJtbKyUr///ntVVVX15MmTKqCGh4cb2/zxxx+qoijqtWvXyjz2iqqs8hAYGKgOHz680LKS/JJWdXeb306dOqnjx4+/5esbNmxQNRpNoT/ICxYsUB0dHdXs7Owyib0yKKv3748//qhaWlqqubm5xmXV9f0bFBSkjh071vhcr9ervr6+6v/+979i2z/55JNqnz59Ci0LDg5WX3jhBVVVS/Z5XJ2UNr//lZeXpzo4OKjLly83LgsJCVH79+9f1qFWSqXN752+V8j7t7B7ff/Onj1bdXBwUNPS0ozL5P1bvJL8DZo0aZLatGnTQssGDRqk9uzZ0/j8Xn9mpiBd1+5RZGQk0dHRdOvWzbjMycmJ4OBg9u7dC8DevXtxdnambdu2xjbdunVDo9EQGhpa7jGbS1nk4cCBAxw+fJjnn3++yGtjx47F3d2doKAglixZglrN5sK9l/x+9913uLu706xZMyZPnkxGRkah7TZv3hwvLy/jsp49e5KSksKJEyfK/kAqqLL6PU5OTsbR0RELC4tCy6vb+zcnJ4cDBw4U+uzUaDR069bN+Nn5X3v37i3UHvLfiwXtS/J5XF3cTX7/KyMjg9zcXFxdXQst37FjB56enjRs2JAxY8Zw48aNMo29Mrjb/KalpeHv74+fnx/9+/cv9Bkq799/lcX7d/HixTz11FPY2dkVWi7v37tzp8/fsviZmYLFnZuI24mOjgYo9CWw4HnBa9HR0Xh6ehZ63cLCAldXV2Ob6qAs8rB48WIaN25Mx44dCy2fMWMGXbt2xdbWls2bN/Piiy+SlpbGyy+/XGbxV3R3m9/Bgwfj7++Pr68vR48e5Y033uD06dOsWbPGuN3i3t8Fr1UXZfH+jY+P57333mPUqFGFllfH9298fDx6vb7Y99apU6eKXedW78WbP2sLlt2qTXVxN/n9rzfeeANfX99CX1wefvhhBg4cSO3atTl//jxvvfUWvXr1Yu/evWi12jI9horsbvLbsGFDlixZQosWLUhOTuaTTz6hY8eOnDhxgpo1a8r79yb3+v4NCwvj+PHjLF68uNByef/evVt9/qakpJCZmUliYuI9f+aYQrUodN58800++uij27aJiIigUaNG5RRR1VLS/N6rzMxMVq5cybvvvlvktZuXtWrVivT0dGbOnFklviiaOr83f+lu3rw5Pj4+PPTQQ5w/f566deve9XYri/J6/6akpNCnTx+aNGnCtGnTCr1Wld+/onL68MMPWbVqFTt27Ch0w/xTTz1l/H/z5s1p0aIFdevWZceOHTz00EPmCLXS6NChAx06dDA+79ixI40bN+bLL7/kvffeM2NkVc/ixYtp3rw5QUFBhZbL+7f6qRaFzquvvsqwYcNu26ZOnTp3tW1vb28AYmJi8PHxMS6PiYkhMDDQ2CY2NrbQenl5eSQkJBjXr8xKmt97zcPPP/9MRkYGQ4cOvWPb4OBg3nvvPbKzs7Gysrpj+4qsvPJbIDg4GIBz585Rt25dvL29i4yaEhMTAyDv3xLmNzU1lYcffhgHBwd++eUXdDrdbdtXpffvrbi7u6PVao3vpQIxMTG3zKe3t/dt25fk87i6uJv8Fvjkk0/48MMP2bp1Ky1atLht2zp16uDu7s65c+eq1RfFe8lvAZ1OR6tWrTh37hwg79+b3Ut+09PTWbVqFTNmzLjjfqrr+/du3Orz19HRERsbG7Ra7T3/TphCtbhHx8PDg0aNGt32YWlpeVfbrl27Nt7e3mzbts24LCUlhdDQUOOZmw4dOpCUlMSBAweMbbZv347BYDB+qazMSprfe83D4sWLeeSRR/Dw8Lhj28OHD+Pi4lIlviSWV34LHD58GMD4h7ZDhw4cO3as0Jf8LVu24OjoSJMmTcrmIM3I1PlNSUmhR48eWFpa8uuvvxYZTrY4Ven9eyuWlpa0adOm0GenwWBg27Zthc5636xDhw6F2kP+e7GgfUk+j6uLu8kvwMcff8x7773Hxo0bC92PditXr17lxo0bhb6YVwd3m9+b6fV6jh07ZsydvH//dS/5/emnn8jOzuaZZ565436q6/v3btzp87csfidMwmzDIFRQly5dUg8dOmQcwvjQoUPqoUOHCg1l3LBhQ3XNmjXG5x9++KHq7Oysrlu3Tj169Kjav3//YoeXbtWqlRoaGqru2rVLrV+/frUdXvp2ebh69arasGFDNTQ0tNB6Z8+eVRVFUf/4448i2/z111/Vr776Sj127Jh69uxZdf78+aqtra06ZcoUkx9PRVPa/J47d06dMWOGun//fjUyMlJdt26dWqdOnf9v7+5dWgejOI5H0ESxiIilkxZEXUQsDrpIOwgF/QfUoQQHnQUt1EGKm4O4iODoKOIqviB0El18WbQKStTdQQrioP7ucptLbnv1Ziok3w90aU4e6MnhSc7QEyWTSfec8njpdDqtq6srHRwcKBqNhna8tJ/8vr6+anh4WP39/bq/v/eMNP34+JAU7vrd3t6WZVna2trSzc2NZmdn1dra6k74y2QyyuVybvzJyYnq6+u1urqqYrGofD5fdbz0T/txWPjN78rKikzT1O7urqdWy/e/UqmkhYUFnZ6eynEcHR8fa3BwUD09PXp/f6/Jb6wlv/ldXl7W4eGhHh4edH5+rsnJSTU2Nur6+tqNoX7/8JvfspGREU1MTFR8T/16lUol9xnXMAytra3p8vJST09PkqRcLqdMJuPGl8dLZ7NZFYtFbWxsVB0v/d01qwUanb/Yti3DMCo+hULBjTF+v/Oi7OvrS0tLS4rFYrIsS6Ojo7q7u/Os+/LyoqmpKUUiEbW0tGh6etrTPIXFT3lwHKci35K0uLiojo4OfX5+Vqy5v7+vRCKhSCSi5uZmDQwMaHNzs2ps0PnN7/Pzs5LJpNra2mRZlrq7u5XNZj3v0ZGkx8dHjY2NqampSe3t7Zqfn/eMRw4Lv/ktFApV9xPDMOQ4jiTqd319XZ2dnTJNU0NDQzo7O3OPpVIp2bbtid/Z2VFvb69M01RfX5/29vY8x/9nPw4TP/mNx+NVazWfz0uS3t7elE6nFY1G1dDQoHg8rpmZmZo+xNSan/zOzc25sbFYTOPj47q4uPCsR/16+d0fbm9vZRiGjo6OKtaifr3+dX8q59S2baVSqYpzEomETNNUV1eX51m47LtrVgt1UsBnmAIAAAAInVD8RwcAAABAuNDoAAAAAAgcGh0AAAAAgUOjAwAAACBwaHQAAAAABA6NDgAAAIDAodEBAAAAEDg0OgAAAAACh0YHAAAAQODQ6AAAAAAIHBodAAAAAIHzC/TPZ1Y58IjYAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As you can see the difference between the 8-bit and the original values are minimal. But... we need to use the 4 bit quantization if we want to load the 7B Model into a 16GB GPU without problems."
      ],
      "metadata": {
        "id": "CLv-uYtlT2Bo"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "620MdVMk7iUS"
      },
      "source": [
        "\n",
        "# QLoRA. Train with LoRA a quantized LLM Model.\n",
        "We are going to train with LoRA a Bloom 7B Model Quantizated to 4 bits."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load the PEFT and Datasets Libraries.\n",
        "\n",
        "The PEFT library contains the Hugging Face implementation of differente fine-tuning techniques, like LoRA Tuning.\n",
        "\n",
        "Using the Datasets library we have acces to a huge amount of Datasets."
      ],
      "metadata": {
        "id": "uml3wgdri2_X"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "_UyyuMGnCPjA"
      },
      "outputs": [],
      "source": [
        "!pip -q install accelerate bitsandbytes\n",
        "!pip -q install datasets\n",
        "!pip -q install trl"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "I'm going to downlad the peft and Transformers library from guthub. Obtaining the lastest version of these libraries we can obtain support for the lastest Models like Llama 2 or Mistral."
      ],
      "metadata": {
        "id": "XuYNBSmTbvtB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Install the lastest versions of peft & transformers library recommended\n",
        "#if you want to work with the most recent models\n",
        "!pip install -q git+https://github.com/huggingface/peft.git\n",
        "!pip install -q git+https://github.com/huggingface/transformers.git"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VVe5LcY9deQ5",
        "outputId": "3bd933c4-4277-4ed4-8012-7f8b9a564cd3"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VOnJlBY-81Wl"
      },
      "source": [
        "From the transformers library we import the necesary classes to import the model and the tokenizer.\n",
        "\n",
        "Then we can load the Tokenizer and the model.\n",
        "\n",
        "Bloom is one of the smarters models available to be trained with PEFT Library using LoRA Tuning. You can use either of the models in the Bloom Family, I encorage you to use at least two of them and see the differences.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
        "from trl import SFTTrainer\n",
        "import torch"
      ],
      "metadata": {
        "id": "pBOE-h8sbrNK"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Hugging Face login"
      ],
      "metadata": {
        "id": "1Bc5eKi_efxO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Model"
      ],
      "metadata": {
        "id": "cwAiEFifgp3-"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "vziwd2UuCYGl"
      },
      "outputs": [],
      "source": [
        "#Use any model you want, if you want to do some fast test, just use the smallest one.\n",
        "\n",
        "#model_name = \"bigscience/bloomz-560m\"\n",
        "#model_name=\"bigscience/bloom-1b1\"\n",
        "model_name = \"bigscience/bloom-7b1\"\n",
        "target_modules = [\"query_key_value\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "With the ***BitsAndBytesConfig*** class of the library ***transfomers*** we can create the configuration needed to load the quantized version of the model.\n",
        "\n",
        "This configuration needs to be sended to the ***AutoModelForCasualLM*** in the ***quantization_config*** parameter."
      ],
      "metadata": {
        "id": "UVF_hKiPd1lh"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "3h_ydWGf6EAd"
      },
      "outputs": [],
      "source": [
        "bnb_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    bnb_4bit_compute_dtype=torch.bfloat16\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177,
          "referenced_widgets": [
            "81d01953e58f4964850dbb745eb9609f",
            "4ee3fdace5a04c809d6c8f526d9b860e",
            "8e3a1dfb817a4712b1e9a18aef90d105",
            "28cc0f6e386d4dec98b31736e7ff0ebe",
            "1d16b8b80b76420cb406252d95cffbf3",
            "c62c6c8e99b44d858062b137116ce980",
            "73f33f3622bd42f4849278a9131466a6",
            "3b7c766ef9dc4dfeb0ab622ab1b50837",
            "61be2cf32eb74af6b45e237cd5d54e16",
            "3e31c5f18f52409f8e3c6ff88f22df1b",
            "6debfbe97496434aa2ff9eb85c07ddad",
            "2336e6e73a10498c9e28f15cf7a34cfc",
            "aa7a8d4b22c34cd6927e255681ab768b",
            "11b8fc9ab9554156b4488c81fe811548",
            "e5223dc4592845cb90dcb65d84548dcd",
            "508810a6e5ac49868ab278d8552d54e8",
            "ef6e3d0914d448ef8c00aebf232f1fa6",
            "83ee78ee868f4348ab294512eccf1768",
            "329f39dd7e8e45b3bb241b48d523cbca",
            "64f0c7defb73476fb4902560713ee4d8",
            "5f09e05e1f1d40f99adb4b4427331a80",
            "08fc1231c0ad4c1fb42b4cd05c637dc8",
            "0c320badc05f46c496c4a1595eb7125e",
            "9848d772bf4c4460889ed5cb14671c26",
            "3098daed65d342379d9a4dc6c18fd503",
            "2d307690a804488ea1f0cdc97c7c65a1",
            "4fdbfabf835e4af5b6532b4dd12b52e7",
            "e55fed8180984a38a212a7c790c941eb",
            "56251681d3b94d7fbdebe607661483eb",
            "7e1a52e4de7a4d81bba8dd3a97fba4db",
            "41254e118e1646698bf731f4937d8918",
            "cc595943252d49bfbc782b73a9a44730",
            "2b4bfcb5a6a7456e9f2a248ffe329d36",
            "b3475a6b14a4424088e21e387820079e",
            "b9f850f328b349a0895f10bda83a2ff7",
            "1fd087fd570944a797071285692f2647",
            "d8711e11d9db4ac491e3ee9dd2ca1db0",
            "1c6026a2b66947498060ed8ccc8ef0e1",
            "45462635fa0741feb8ce2eb6b67b38d3",
            "377e56ab071c4d6e834a3fa796f891ac",
            "1a8a9bb527324552980d90f235bef614",
            "7091d9b93baa43a0a37407d8307610da",
            "1c55292f1cbf478d9968babaa55f8766",
            "13a6927f3a2c4fb9954cf864f713aab4",
            "bf5c00da99194448a2d39a4db6390f26",
            "32ba3bf37e524b9790d379870d6bb4cb",
            "7438ee67299844c0977c5a1f0ff9df30",
            "08a8b4d2684c44598c48da2abe51a0bb",
            "c90eedf858e845f1b25755bbf6606189",
            "6ab530b2f6b14eeaba6dff9f672b87a8",
            "97f004b624d84c6cb29a50c8f7305101",
            "5e58dc2e76654cd08313f9cb10ff3206",
            "c332a7181b874460b3a77bf7fafccd14",
            "d26b825c1c4848b5bf755d3a48329de7",
            "09cd62cb0d16464c84673a27a064c428"
          ]
        },
        "id": "W2EZhNQ66EAd",
        "outputId": "da7e1408-1cb1-456b-eaf1-80d04f7b226e"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)model.bin.index.json:   0%|          | 0.00/27.5k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "81d01953e58f4964850dbb745eb9609f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2336e6e73a10498c9e28f15cf7a34cfc"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)l-00001-of-00002.bin:   0%|          | 0.00/9.98G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0c320badc05f46c496c4a1595eb7125e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)l-00002-of-00002.bin:   0%|          | 0.00/4.16G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b3475a6b14a4424088e21e387820079e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bf5c00da99194448a2d39a4db6390f26"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "device_map = {\"\": 0}\n",
        "foundation_model = AutoModelForCausalLM.from_pretrained(model_name,\n",
        "                    quantization_config=bnb_config,\n",
        "                    device_map=device_map,\n",
        "                    use_cache = False)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we have the quantized version of the model in memory. Try to load the unquantized version see if it's possible."
      ],
      "metadata": {
        "id": "nVDiZYbee77R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "tokenizer.pad_token = tokenizer.eos_token"
      ],
      "metadata": {
        "id": "aU0awofs84q7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qtc1gbK39Hp7"
      },
      "source": [
        "## Inference with the pre-trained model.\n",
        "I'm going to do a test with the pre-trained model without fine-tuning, to see if something changes after the fine-tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jak6FzpvFTHk"
      },
      "outputs": [],
      "source": [
        "#this function returns the outputs from the model received, and inputs.\n",
        "def get_outputs(model, inputs, max_new_tokens=100):\n",
        "    outputs = model.generate(\n",
        "        input_ids=inputs[\"input_ids\"],\n",
        "        attention_mask=inputs[\"attention_mask\"],\n",
        "        max_new_tokens=max_new_tokens,\n",
        "        repetition_penalty=1.5, #Avoid repetition.\n",
        "        early_stopping=False, #The model can stop before reach the max_length\n",
        "        eos_token_id=tokenizer.eos_token_id,\n",
        "    )\n",
        "    return outputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkFqjS459jAa"
      },
      "source": [
        "The dataset used for the fine-tuning contains prompts to be used with Large Language Models.\n",
        "\n",
        "I'm going to request the pre-trained model that acts like a motivational coach."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3BAYg7czFYeK",
        "outputId": "8a3fe63c-04d6-4854-a643-4e35e8cda591"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[\"I want you to act as a motivational coach.  I don't mean that in the sense of telling people what they should do, but rather encouraging them and helping motivate their own actions.\\nYou can start by asking questions like these:\\n\\nWhat are your goals?\\nHow will this help achieve those?\\n\\nThen\"]\n"
          ]
        }
      ],
      "source": [
        "#Inference original model\n",
        "input_sentences = tokenizer(\"I want you to act as a motivational coach. \", return_tensors=\"pt\").to('cuda')\n",
        "foundational_outputs_sentence = get_outputs(foundation_model, input_sentences, max_new_tokens=50)\n",
        "\n",
        "print(tokenizer.batch_decode(foundational_outputs_sentence, skip_special_tokens=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KQUGY47p9ysI"
      },
      "source": [
        "The answer is good enough, Bloom 7B is a really well trained Model. But we will try to improve the quality with a sort fine-tuning process.\n",
        "\n",
        "In a previous Notebook I used the Bloom 560m and the answer was really different. You can check here: https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/5-Fine%20Tuning/LoRA_Tuning_PEFT.ipynb\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FL5L_DcR9ggA"
      },
      "source": [
        "## Preparing the Dataset.\n",
        "The Dataset useds is:\n",
        "\n",
        "https://huggingface.co/datasets/fka/awesome-chatgpt-prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "id": "DyIMQ7IHFbIx",
        "outputId": "07b17cc0-2e5b-4562-a2df-bd21f5bd184f"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['prompt', 'input_ids', 'attention_mask'],\n",
              "    num_rows: 50\n",
              "})"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "dataset = \"fka/awesome-chatgpt-prompts\"\n",
        "\n",
        "#Create the Dataset to create prompts.\n",
        "data = load_dataset(dataset)\n",
        "\n",
        "data = data.map(lambda samples: tokenizer(samples[\"prompt\"]), batched=True)\n",
        "train_sample = data[\"train\"].select(range(50))\n",
        "\n",
        "del data\n",
        "train_sample = train_sample.remove_columns('act')\n",
        "\n",
        "display(train_sample)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gmlZY3fk_9fm",
        "outputId": "3bedb4c0-3f7c-4b1e-eff3-dbf06701770c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'prompt': ['I want you to act as a linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. do not write explanations. do not type commands unless I instruct you to do so. when i need to tell you something in english, i will do so by putting text inside curly brackets {like this}. my first command is pwd'], 'input_ids': [[44, 4026, 1152, 427, 1769, 661, 267, 104105, 28434, 17, 473, 2152, 4105, 49123, 530, 1152, 2152, 57502, 1002, 3595, 368, 28434, 3403, 6460, 17, 473, 4026, 1152, 427, 3804, 57502, 1002, 368, 28434, 10014, 14652, 2592, 19826, 4400, 10973, 15, 530, 16915, 4384, 17, 727, 1130, 11602, 184637, 17, 727, 1130, 4105, 49123, 35262, 473, 32247, 1152, 427, 727, 1427, 17, 3262, 707, 3423, 427, 13485, 1152, 7747, 361, 170205, 15, 707, 2152, 727, 1427, 1331, 55385, 5484, 14652, 6291, 999, 117805, 731, 29726, 1119, 96, 17, 2670, 3968, 9361, 632, 269, 42512]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}\n"
          ]
        }
      ],
      "source": [
        "print(train_sample[:1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVPAJsrUAHiJ"
      },
      "source": [
        "## Fine-Tuning.\n",
        "Firtst is necesary create a LoRA config.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uCalslQFGL7K"
      },
      "outputs": [],
      "source": [
        "# TARGET_MODULES\n",
        "# https://github.com/huggingface/peft/blob/39ef2546d5d9b8f5f8a7016ec10657887a867041/src/peft/utils/other.py#L220\n",
        "\n",
        "import peft\n",
        "from peft import LoraConfig, get_peft_model\n",
        "\n",
        "lora_config = LoraConfig(\n",
        "    r=16, #As bigger the R bigger the parameters to train.\n",
        "    lora_alpha=16, # a scaling factor that adjusts the magnitude of the weight matrix. It seems that as higher more weight have the new training.\n",
        "    target_modules=target_modules,\n",
        "    lora_dropout=0.05, #Helps to avoid Overfitting.\n",
        "    bias=\"none\", # this specifies if the bias parameter should be trained.\n",
        "    task_type=\"CAUSAL_LM\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LUddynl0B1Ck"
      },
      "source": [
        "The most important parameter is **r**, it defines how many parameters will be trained. As bigger the value more parameters are trained, but it means that the model will be able to learn more complicated relations between inputs and outputs.\n",
        "\n",
        "Yo can find a list of the **target_modules** available on the [Hugging Face Documentation]( https://github.com/huggingface/peft/blob/39ef2546d5d9b8f5f8a7016ec10657887a867041/src/peft/utils/other.py#L220)\n",
        "\n",
        "**lora_alpha**. Ad bigger the number more weight have the LoRA activations, it means that the fine-tuning process will have more impac as bigger is this value.\n",
        "\n",
        "**lora_dropout** is like the commom dropout is used to avoid overfitting.\n",
        "\n",
        "**bias** I was hesitating if use *none* or *lora_only*. For text classification the most common value is none, and for chat or question answering, *all* or *lora_only*.\n",
        "\n",
        "**task_type**. Indicates the task the model is beign trained for. In this case, text generation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HArPQ_lvGUkY"
      },
      "outputs": [],
      "source": [
        "#Create a directory to contain the Model\n",
        "import os\n",
        "working_dir = './'\n",
        "\n",
        "output_directory = os.path.join(working_dir, \"peft_lab_outputs\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rWalmqWm4STo"
      },
      "source": [
        "In the TrainingArgs we inform the number of epochs we want to train, the output directory and the learning_rate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ND0aJ-t6ARqD"
      },
      "outputs": [],
      "source": [
        "#Creating the TrainingArgs\n",
        "import transformers\n",
        "from transformers import TrainingArguments # , Trainer\n",
        "training_args = TrainingArguments(\n",
        "    output_dir=output_directory,\n",
        "    auto_find_batch_size=True, # Find a correct bvatch size that fits the size of Data.\n",
        "    learning_rate= 2e-4, # Higher learning rate than full fine-tuning.\n",
        "    num_train_epochs=5\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OgxsV-iy_J_o"
      },
      "source": [
        "Now we can train the model.\n",
        "To train the model we need:\n",
        "\n",
        "\n",
        "*   The Model.\n",
        "*   The training_args\n",
        "* The Dataset\n",
        "* The result of DataCollator, the Dataset ready to be procesed in blocks.\n",
        "* The LoRA config.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 220
        },
        "id": "z5NYHqBnGZyF",
        "outputId": "30dc63b0-8e9d-4fdb-e248-8a195f2767f8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:173: UserWarning: You didn't pass a `max_seq_length` argument to the SFTTrainer, this will default to 1024\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:214: UserWarning: You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code.\n",
            "  warnings.warn(\n",
            "You're using a BloomTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='65' max='65' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [65/65 06:37, Epoch 5/5]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TrainOutput(global_step=65, training_loss=2.7377777099609375, metrics={'train_runtime': 404.0462, 'train_samples_per_second': 0.619, 'train_steps_per_second': 0.161, 'total_flos': 966262938697728.0, 'train_loss': 2.7377777099609375, 'epoch': 5.0})"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "trainer = SFTTrainer(\n",
        "    model=foundation_model,\n",
        "    args=training_args,\n",
        "    train_dataset=train_sample,\n",
        "    peft_config = lora_config,\n",
        "    dataset_text_field=\"prompt\",\n",
        "    tokenizer=tokenizer,\n",
        "    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)\n",
        ")\n",
        "trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kEKiFdpDGgOx"
      },
      "outputs": [],
      "source": [
        "#Save the model.\n",
        "peft_model_path = os.path.join(output_directory, f\"lora_model\")\n",
        "\n",
        "trainer.model.save_pretrained(peft_model_path)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#I'm going to clean some variables just to avoid memory problems\n",
        "import gc\n",
        "import torch\n",
        "del foundation_model\n",
        "del trainer\n",
        "del train_sample\n",
        "torch.cuda.empty_cache()\n",
        "gc.collect()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "glDE0Z4FOSe0",
        "outputId": "b05dd514-c258-4652-91f6-1d2b5903304b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "33"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inference with the pretrained model"
      ],
      "metadata": {
        "id": "UOAqEg0mSjHW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#import peft\n",
        "from peft import AutoPeftModelForCausalLM, PeftConfig\n",
        "#import os\n",
        "\n",
        "device_map = {\"\": 0}\n",
        "working_dir = './'\n",
        "\n",
        "output_directory = os.path.join(working_dir, \"peft_lab_outputs\")\n",
        "peft_model_path = os.path.join(output_directory, f\"lora_model\")\n"
      ],
      "metadata": {
        "id": "dX5d8xMCSC6y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_TAjrSWSe14q",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "18a4c08b42fc4da7a362a7884cb3cd22",
            "9ef395f96f1f420fbec3509b7970b6be",
            "d65faa314a4647ca996e32719f1274e6",
            "fee7bd3e5f8240fc9fcf252322295fd5",
            "7542b49fcf6f4daa82dff553fd1397c5",
            "8dc80d0f55d0465299528c0a72556b6a",
            "a2a99223a1844458b1cfe503ff31e411",
            "cec5ebb9f6834465976662bca7b30fc5",
            "f29fd7473dd94da1b9de225458921ef7",
            "1665d50b76df47569c73b390d14f8cff",
            "61ae660bd0c94c7e81a0a4e4619122f0"
          ]
        },
        "outputId": "9cb8d434-82f9-4a23-95e1-98040d42fd20"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "18a4c08b42fc4da7a362a7884cb3cd22"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "#Load the Model.\n",
        "loaded_model = AutoPeftModelForCausalLM.from_pretrained(\n",
        "                                        peft_model_path,\n",
        "                                        torch_dtype=torch.bfloat16,\n",
        "                                        is_trainable=False,\n",
        "                                        load_in_4bit=True,\n",
        "                                        device_map = 'auto')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dK--YFPR6OxH"
      },
      "source": [
        "## Inference the fine-tuned model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I_27uvJudf03",
        "outputId": "bc6d4b5f-61e6-482b-987f-d4f9a5930dd3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[\"I want you to act as a motivational coach.  You will be working with an individual who is struggling in their career and has not been able find success. The person may have had some previous experience, but they are now looking for new opportunities that can help them achieve more.\\nThe client's current situation\"]\n"
          ]
        }
      ],
      "source": [
        "input_sentences = tokenizer(\"I want you to act as a motivational coach. \", return_tensors=\"pt\").to('cuda')\n",
        "foundational_outputs_sentence = get_outputs(loaded_model, input_sentences, max_new_tokens=50)\n",
        "\n",
        "print(tokenizer.batch_decode(foundational_outputs_sentence, skip_special_tokens=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCV9JOBG6Ug8"
      },
      "source": [
        "The result is really good. Let's compare the answer of the pre-trained Model with the fine-tuned one:\n",
        "* **Pretrained Model:** I want you to act as a motivational coach.  I don't mean that in the sense of telling people what they should do, but rather encouraging them and helping motivate their own actions.\\nYou can start by asking questions like these:\\n\\nWhat are your goals?\\nHow will this help achieve those?\\n\\nThen.\n",
        "* **Fine-Tuned Model:** I want you to act as a motivational coach.  I will provide some details about an individual who needs help improving their confidence, and your goal is  \"Ideas for helping someone improve self-confidence.\" Your first suggestion should be \"Provide encouragement when they need it most\"; my reply\n",
        "\n",
        "As you can see the result is really similar to the samples contained in the Datased used to fine-tune the Model. And we only trained the Model for some epochs and with a really small number of rows.\n",
        "\n",
        "# Continue Learning\n",
        "Please, play with all the variables in the notebook and drive your own experiments and get your conclusions. Changes in the ***lora_alpha*** can have a big effect in the results.\n",
        "\n",
        "You can change the rest of **lora_config** values, maybe you can achieve a better result in less epochs, saving time and money for your company. :-)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gyTM62Tsg8oX"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "18a4c08b42fc4da7a362a7884cb3cd22": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9ef395f96f1f420fbec3509b7970b6be",
              "IPY_MODEL_d65faa314a4647ca996e32719f1274e6",
              "IPY_MODEL_fee7bd3e5f8240fc9fcf252322295fd5"
            ],
            "layout": "IPY_MODEL_7542b49fcf6f4daa82dff553fd1397c5"
          }
        },
        "9ef395f96f1f420fbec3509b7970b6be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8dc80d0f55d0465299528c0a72556b6a",
            "placeholder": "​",
            "style": "IPY_MODEL_a2a99223a1844458b1cfe503ff31e411",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "d65faa314a4647ca996e32719f1274e6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cec5ebb9f6834465976662bca7b30fc5",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f29fd7473dd94da1b9de225458921ef7",
            "value": 2
          }
        },
        "fee7bd3e5f8240fc9fcf252322295fd5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1665d50b76df47569c73b390d14f8cff",
            "placeholder": "​",
            "style": "IPY_MODEL_61ae660bd0c94c7e81a0a4e4619122f0",
            "value": " 2/2 [00:18&lt;00:00,  8.40s/it]"
          }
        },
        "7542b49fcf6f4daa82dff553fd1397c5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8dc80d0f55d0465299528c0a72556b6a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a2a99223a1844458b1cfe503ff31e411": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cec5ebb9f6834465976662bca7b30fc5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f29fd7473dd94da1b9de225458921ef7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1665d50b76df47569c73b390d14f8cff": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61ae660bd0c94c7e81a0a4e4619122f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "81d01953e58f4964850dbb745eb9609f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4ee3fdace5a04c809d6c8f526d9b860e",
              "IPY_MODEL_8e3a1dfb817a4712b1e9a18aef90d105",
              "IPY_MODEL_28cc0f6e386d4dec98b31736e7ff0ebe"
            ],
            "layout": "IPY_MODEL_1d16b8b80b76420cb406252d95cffbf3"
          }
        },
        "4ee3fdace5a04c809d6c8f526d9b860e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c62c6c8e99b44d858062b137116ce980",
            "placeholder": "​",
            "style": "IPY_MODEL_73f33f3622bd42f4849278a9131466a6",
            "value": "Downloading (…)model.bin.index.json: 100%"
          }
        },
        "8e3a1dfb817a4712b1e9a18aef90d105": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3b7c766ef9dc4dfeb0ab622ab1b50837",
            "max": 27460,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_61be2cf32eb74af6b45e237cd5d54e16",
            "value": 27460
          }
        },
        "28cc0f6e386d4dec98b31736e7ff0ebe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e31c5f18f52409f8e3c6ff88f22df1b",
            "placeholder": "​",
            "style": "IPY_MODEL_6debfbe97496434aa2ff9eb85c07ddad",
            "value": " 27.5k/27.5k [00:00&lt;00:00, 1.58MB/s]"
          }
        },
        "1d16b8b80b76420cb406252d95cffbf3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c62c6c8e99b44d858062b137116ce980": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "73f33f3622bd42f4849278a9131466a6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3b7c766ef9dc4dfeb0ab622ab1b50837": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61be2cf32eb74af6b45e237cd5d54e16": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3e31c5f18f52409f8e3c6ff88f22df1b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6debfbe97496434aa2ff9eb85c07ddad": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2336e6e73a10498c9e28f15cf7a34cfc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_aa7a8d4b22c34cd6927e255681ab768b",
              "IPY_MODEL_11b8fc9ab9554156b4488c81fe811548",
              "IPY_MODEL_e5223dc4592845cb90dcb65d84548dcd"
            ],
            "layout": "IPY_MODEL_508810a6e5ac49868ab278d8552d54e8"
          }
        },
        "aa7a8d4b22c34cd6927e255681ab768b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ef6e3d0914d448ef8c00aebf232f1fa6",
            "placeholder": "​",
            "style": "IPY_MODEL_83ee78ee868f4348ab294512eccf1768",
            "value": "Downloading shards: 100%"
          }
        },
        "11b8fc9ab9554156b4488c81fe811548": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_329f39dd7e8e45b3bb241b48d523cbca",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_64f0c7defb73476fb4902560713ee4d8",
            "value": 2
          }
        },
        "e5223dc4592845cb90dcb65d84548dcd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5f09e05e1f1d40f99adb4b4427331a80",
            "placeholder": "​",
            "style": "IPY_MODEL_08fc1231c0ad4c1fb42b4cd05c637dc8",
            "value": " 2/2 [00:46&lt;00:00, 21.48s/it]"
          }
        },
        "508810a6e5ac49868ab278d8552d54e8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ef6e3d0914d448ef8c00aebf232f1fa6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "83ee78ee868f4348ab294512eccf1768": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "329f39dd7e8e45b3bb241b48d523cbca": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "64f0c7defb73476fb4902560713ee4d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "5f09e05e1f1d40f99adb4b4427331a80": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "08fc1231c0ad4c1fb42b4cd05c637dc8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0c320badc05f46c496c4a1595eb7125e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9848d772bf4c4460889ed5cb14671c26",
              "IPY_MODEL_3098daed65d342379d9a4dc6c18fd503",
              "IPY_MODEL_2d307690a804488ea1f0cdc97c7c65a1"
            ],
            "layout": "IPY_MODEL_4fdbfabf835e4af5b6532b4dd12b52e7"
          }
        },
        "9848d772bf4c4460889ed5cb14671c26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e55fed8180984a38a212a7c790c941eb",
            "placeholder": "​",
            "style": "IPY_MODEL_56251681d3b94d7fbdebe607661483eb",
            "value": "Downloading (…)l-00001-of-00002.bin: 100%"
          }
        },
        "3098daed65d342379d9a4dc6c18fd503": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7e1a52e4de7a4d81bba8dd3a97fba4db",
            "max": 9976268775,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_41254e118e1646698bf731f4937d8918",
            "value": 9976268775
          }
        },
        "2d307690a804488ea1f0cdc97c7c65a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cc595943252d49bfbc782b73a9a44730",
            "placeholder": "​",
            "style": "IPY_MODEL_2b4bfcb5a6a7456e9f2a248ffe329d36",
            "value": " 9.98G/9.98G [00:32&lt;00:00, 384MB/s]"
          }
        },
        "4fdbfabf835e4af5b6532b4dd12b52e7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e55fed8180984a38a212a7c790c941eb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "56251681d3b94d7fbdebe607661483eb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7e1a52e4de7a4d81bba8dd3a97fba4db": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "41254e118e1646698bf731f4937d8918": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cc595943252d49bfbc782b73a9a44730": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2b4bfcb5a6a7456e9f2a248ffe329d36": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b3475a6b14a4424088e21e387820079e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b9f850f328b349a0895f10bda83a2ff7",
              "IPY_MODEL_1fd087fd570944a797071285692f2647",
              "IPY_MODEL_d8711e11d9db4ac491e3ee9dd2ca1db0"
            ],
            "layout": "IPY_MODEL_1c6026a2b66947498060ed8ccc8ef0e1"
          }
        },
        "b9f850f328b349a0895f10bda83a2ff7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_45462635fa0741feb8ce2eb6b67b38d3",
            "placeholder": "​",
            "style": "IPY_MODEL_377e56ab071c4d6e834a3fa796f891ac",
            "value": "Downloading (…)l-00002-of-00002.bin: 100%"
          }
        },
        "1fd087fd570944a797071285692f2647": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1a8a9bb527324552980d90f235bef614",
            "max": 4161876233,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7091d9b93baa43a0a37407d8307610da",
            "value": 4161876233
          }
        },
        "d8711e11d9db4ac491e3ee9dd2ca1db0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1c55292f1cbf478d9968babaa55f8766",
            "placeholder": "​",
            "style": "IPY_MODEL_13a6927f3a2c4fb9954cf864f713aab4",
            "value": " 4.16G/4.16G [00:13&lt;00:00, 280MB/s]"
          }
        },
        "1c6026a2b66947498060ed8ccc8ef0e1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "45462635fa0741feb8ce2eb6b67b38d3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "377e56ab071c4d6e834a3fa796f891ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1a8a9bb527324552980d90f235bef614": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7091d9b93baa43a0a37407d8307610da": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1c55292f1cbf478d9968babaa55f8766": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "13a6927f3a2c4fb9954cf864f713aab4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bf5c00da99194448a2d39a4db6390f26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_32ba3bf37e524b9790d379870d6bb4cb",
              "IPY_MODEL_7438ee67299844c0977c5a1f0ff9df30",
              "IPY_MODEL_08a8b4d2684c44598c48da2abe51a0bb"
            ],
            "layout": "IPY_MODEL_c90eedf858e845f1b25755bbf6606189"
          }
        },
        "32ba3bf37e524b9790d379870d6bb4cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6ab530b2f6b14eeaba6dff9f672b87a8",
            "placeholder": "​",
            "style": "IPY_MODEL_97f004b624d84c6cb29a50c8f7305101",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "7438ee67299844c0977c5a1f0ff9df30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5e58dc2e76654cd08313f9cb10ff3206",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c332a7181b874460b3a77bf7fafccd14",
            "value": 2
          }
        },
        "08a8b4d2684c44598c48da2abe51a0bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d26b825c1c4848b5bf755d3a48329de7",
            "placeholder": "​",
            "style": "IPY_MODEL_09cd62cb0d16464c84673a27a064c428",
            "value": " 2/2 [00:19&lt;00:00,  8.64s/it]"
          }
        },
        "c90eedf858e845f1b25755bbf6606189": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6ab530b2f6b14eeaba6dff9f672b87a8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "97f004b624d84c6cb29a50c8f7305101": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5e58dc2e76654cd08313f9cb10ff3206": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c332a7181b874460b3a77bf7fafccd14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d26b825c1c4848b5bf755d3a48329de7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "09cd62cb0d16464c84673a27a064c428": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}